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Executive summary 



The Australian vocational education and training (VET) system has undergone considerable change 
over the last decade, and part of that change is the high priority given to flexible learning and 
delivery. Since the establishment of the National Flexible Delivery Taskforce in 1995, flexible 
delivery and learning for Australian vocational education and training has been supported by a 
range of national initiatives. This has culminated in the development and implementation of the 
Australian Flexible Learning Framework for the national VET system 2000—2004. 

The framework sets out goals and strategies to enable the VET system to: 

V embrace new understandings of learning in every aspect of its operation 

V be accessible at different stages of clients’ lives and have the mechanisms to recognise and 
value people’s experiences as knowledge 

V be delivered through the media appropriate to the client’s learning preference 

V be convenient when balanced against the competing demands in the learner’s life 

V be accessible to clients ‘where, when and how’ they want it. 

Existing and current research 

While there has been considerable research into technological and pedagogical aspects of flexible 
learning design, very few studies have focussed on assessment in flexible delivery and learning 
arrangements. In fact, there are very few publications highlighting the issues of assessment for 
open, distance and flexible learning in the Australian VET sector. 

The broad purpose of this pilot study is to explore and examine assessment delivered in a flexible 
mode in the VET sector in order to determine: 

V the range of assessment methods 

V the pedagogical and technological reasons for the selection of the assessment methods 

V the validity of the approaches taken. 

Australian Quality Training Framework standards 

Assessment that is valid, reliable, fair and flexible is the cornerstone of the Australian VET 
system — flexibility being a key principle of good practice (Australian National Training Authority 
[ANTA] 1996). These principles of good assessment practice are embedded in the 2001 Australian 
Quality Training Framework standards. 

The compliance requirements under this framework highlight the importance of consistency in a 
national VET assessment system. For example, the principle of mutual recognition requires that all 
registered training organisations recognise the qualifications issued by other training providers. 

This has reinforced the need for confidence in the assessment processes for all stakeholders in the 
Australian system. 
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This study involved a review of literature, a survey of providers and 13 case studies of assessment 
practice in a range of VET programs. Key findings from the study relate to the following areas: 

Assessment environments 

Assessment approaches are linked to the delivery mode, the industry, the nature of the 
competencies, the level of the qualification and the learner groups. The role of the workplace or 
enterprise is an important consideration for assessment planning issues. 

Evidence collection 

To support judgements of competence, assessors collect a range of evidence. The full range of 
assessment methods is evident across the sites examined in this project and there are no examples 
of one-off assessment events that occur only at the end of the learning experience (i.e. summative 
assessments only). The nature of the competencies influences the choice of assessment method. In 
a number of cases, the needs of the training package influence the approach taken; in particular, 
where workplace assessment is specified, and workplace and assessors observe performance using 
standard checklists. 

Formative assessment and feedback 

The importance and critical role of ongoing or formative assessment and feedback in flexible 
delivery is acknowledged by many of the informants. Approaches to formative assessment and the 
provision of feedback vary and are influenced by such factors as availability of resources, student 
locations, system infrastructure and educational philosophy. Distance modes using print-based 
resources tend to use written exercises in learning guides, while those using online modes are 
making greater use of computerised self-assessment activities. The study found that written 
responses make up a large part of distance-mode assessment, although this is often combined with 
other forms of assessment, including workplace assessment and oral interviews. Adequate 
provision of feedback and maintaining contact with learners, while identified as a critical factor for 
successful learning and course completion rates in flexible learning modes, are at the same time 
seen as highly time-consuming activities. 

Features of flexibility in assessment 

Features of flexibility in assessment are reflected in the student-teacher negotiation processes 
around the timing of assessment, the location of the assessment, the choice of assessment method, 
the capacity to contextualise assessment tasks to learners’ needs and those of workplaces, and a 
choice of assessment methods to meet special needs. Invigilated ‘supervisor’ verified assessment 
activities are also considered appropriate for summative assessment purposes. 



Recognition of prior learning or current competence 

Approaches and attitudes to recognition of prior learning or current competence as an assessment 
pathway vary. At three of the major case-study sites, the recognition process is well integrated in 
the learning and assessment pathway. This is frequendy a feature where the registered training 
organisation works closely with the enterprise or the learner’s workplace. While the availability of 
recognition of prior learning or current competence options for learners is provided at enrolment 
in the other case-study sites, it is more common for students to enroll and then be ‘fast-tracked’ 
through course assessment requirements. This approach was taken for different reasons: some 
informants mentioned that learners often found the recognition process ‘onerous’ in terms of the 
time and quantity of evidence required, while others suggested that learners often prefer the 
opportunity to take the course as a ‘refresher’ program. The issue of actual student contact hours 
was raised by informants because of its impact on staffing levels and resource allocations. 
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Involvement in assessment 



The involvement of a range of personnel in assessment is part of the complexity of assessment in a 
performance-based process. The nature of some competencies is such that extended periods of 
time are required to confirm competence, and this often involves personnel outside the registered 
training organisation. Testimonials and third-party evidence are increasingly features of assessment 
approaches. 

In more than 50% of the programs surveyed for this study, a person or persons other than the 
teacher or trainer had a role to play in the assessment process. Frequendy this role encompassed 
verifying evidence provided by the learner, or contributing to the judgment of workplace 
performance. 

Another factor relates to the changing levels of engagement of the provider with the various phases 
of assessment design, planning and delivery (known as the assessment ‘chain’). The study noted 
issues in regard to the relationship between assessment designers and practitioners, particularly 
where learning and assessment packages are purchased by the registered training organisation. 
Where assessment design has occurred at a point removed from the delivery of the training and 
assessment, guidelines to the assessor are critical, and explicit information about contextualising the 
assessment needs to be provided. Opportunities to review and adapt assessments are important in 
this context. 



Technology 

Technology is used in a number of ways to support and deliver assessment. Increasingly, email is 
used to communicate with students and for submission of assessments. Computer-assisted 
formative assessment, with a strong focus on multiple choice, true/false and matching type 
questions is also quite extensively used. 



Learner-centred approaches and infrastructure support 

Flexible delivery implies greater individualisation in learning and a stronger emphasis on a learner- 
centred approach. A range of issues was raised by informants in this study. Those of particular 
importance to the assessment processes include: 

open enrolment periods in flexible delivery 

T choice in sequencing of learning activities and the impact of this on collaborative learning and 
assessment 

T multiple input to assessment by a range of personnel 
T tracking learner progress. 

A key consideration for informants is meeting learner expectations in regard to response times. 
Being able to contact learners individually about their assessments is seen as a time consuming but 
necessary activity. 

Lack of time and the need for more resources for assessment are often mentioned, especially in 
relation to the time required for both the review of assessment processes and the maintenance of 
assessment tasks, security and authenticity. 



Conclusions 

The study highlights the diversity of flexible delivery and assessment arrangements across the 
Australian VET sector. While assessment practice in flexible delivery is underpinned by training 
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packages and course curriculum assessment requirements, a range of other factors influences the 
decisions made by teachers and assessors in their selection of assessment methods. These are: 

providing qualified workplace assessors 

<■ ascertaining distance factors and the location of the student 

ensuring evidence provided by students is authentic 

involving the learner in the assessment process 

involving the workplace in the process of assessment through hearing third-party evidence. 

There are particular challenges for assessment in flexible delivery modes. Many of these challenges 
relate to the implementation of effective support arrangements for assessment (for example, timely 
and appropriate feedback, well-designed feedback systems when computer-assisted assessment is 
being used, and regular contact). Other challenges relate specifically to the administrative and 
management issues that emerge when organisations offer a range of learning and assessment 
options (for example, tracking learner progress and ensuring integrity of evidence). Other 
challenges relate more to pedagogical considerations. The use of information communication 
technologies for flexible learning and assessment arrangements has sharpened the interest in 
instructional design, learning theory and the role of assessment in learning. 

The difference in the degree of face-to-face contact between learners, assessors and teachers in 
flexible delivery and ‘traditional’ delivery modes is a real issue. Sound assessment design is required 
at all stages of the development and implementation of flexible delivery to take account of this 
issue. 
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Project background 
and methodology 



Context 

This small pilot study into assessment in flexible delivery models was undertaken in the diverse and 
complex Australian VET sector. 

The sample of flexible delivery case-study sites, small as it is, reflects that diversity and complexity, 
in terms of size, location, scope of delivery and learner groups. VET programs are delivered 
through a range of registered training organisations including small- and large-scale public 
institutions, private colleges, private firms of one or two consultants, enterprises, schools and 
community colleges. Clients of the VET sector are as varied as are the providers; studying full- or 
part-time; currently employed, highly skilled or seeking entry to the workforce. 

The scope of qualifications offered — increased by the introduction of training packages and the 
evolution of industry sectors — is also part of the context for this study. As flexible delivery models 
emerge, a ‘borderless’ VET market develops. The growth in the use of information and 
communication technologies to support delivery and assessment will continue to change VET 
delivery over the next decade. 

This study revealed how difficult it is to describe VET activities as one uniform sector. The growth 
of interest and focus on flexible learning and delivery models, in particular online, suggests both 
challenges and opportunities for practitioners in a competency-based training and assessment 
system. Foremost among these are the design of assessment approaches that can achieve a balance 
between the needs of the individual and industry, and the requirements of a national VET system. 



Research methods 

Research questions 

The study centred on the following questions. 

-y- What assessment methods are applied in flexible learning environments, and what is the role of 
technology in each? 

-y- What role does assessment play in flexible delivery? Is it different from assessment in face-to- 
face programs? 

What are the critical factors that influence the selection of assessment methods in programs 
delivered flexibly? 

-y- What strategies help ensure that assessment meets the needs of flexible learners? 

What mechanisms help ensure the validity and reliability of assessment in programs delivered 
flexibly? 

What constraints influence the flexibility of assessment in these programs? 
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How the research was conducted 

This study was conducted in four stages, using qualitative research methods including: 

desk research (document analysis and web review) which led to a literature review (national and 
international) (Literature review) 

a survey of selected providers (Provider survey) 

<■ descriptive case studies based on structured interviews (Case studies) 

<■ the pilot use of a self-evaluation tool (Self-assessment of practice). 

Pilot-survey data were gathered by fax and email, using both tick boxes for responses and open 
questions (appendix 1). Informants for the case-study sites were involved either in the 
development, facilitation and/or assessment of programs. The project sought examples of flexible 
delivery models demonstrating or combining: 

<■ distance learning, using print-based and related material 

online delivery 

learning which is centre based, using a mix of print-based and computer-assisted learning. 

Stage 1 — research preparation 

Literature review 

A literature review with a national and international focus was conducted at the outset of the study. 
The review covered: 

national VET policy documents and research in assessment and flexible delivery 

-y- VET assessment practice in general 

VET assessment practice within a flexible delivery framework 

<■ theoretical views on assessment in a standards-driven, competency-based system 

-v> experience in other educational or related sectors that might be relevant to the Australian VET 
context. 

The review of literature from other sectors and of international experience is of particular 
importance because of the increasing interest in computer-mediated or assisted assessment. 

A document analysis, web-based review and follow-up consultations then identified a range of 30 
flexibly delivered national courses leading to Australian Qualifications Framework qualifications. 
Given the exploratory nature of the study, convenience and access to information guided sampling 
choices. Examples were sought for a range of: 

provider types 

delivery sites 

discipline and industry areas 

Australian Qualifications Framework levels 

-y- delivery options (for example, distance, face-to-face, centre-based, print-based, wholly online, 
combinations of technology and print-based delivery, and use of an Australian National 
Training Authority [ANTA] toolbox). 



NCVER 



11 




Provider survey 



The pilot provider survey was developed and trialled by educational and flexible delivery 
consultants and workplace assessors. Adjustments were then made in response to feedback. 

Following an audit of potential participants, providers were telephoned to canvas their interest and 
willingness to participate. Forty providers from a range of types, Australian Qualifications 
Framework levels, locations and industry sectors were then sent a letter outlining project details 
and protocols. They were also sent the provider survey form. 

This survey provided the data for a provider matrix (appendix 2) that identifies assessment 
methods used in each case. 

Stage 2 — data collection 

Case-study sites 

Examples of 13 current, flexibly delivered courses, in distance learning, centre-based and online 
mode were then selected from this matrix. The sample was determined by criteria in the project 
brief, with case-study sites chosen from those identified in the first stage. 

Structured interviews elicited detailed information about features of the assessment approach taken 
by each organisation, such as: 

assessment methods 

T assessment information available to candidates 
tr support arrangements for learners in regard to assessment 
<■ strategies and resources available to assessors 

-y- issues and constraints associated with assessment in student induction and in delivery of the 
course. 

Respondents were asked to identify issues that influenced the choice of assessment methods and, 
where relevant, the effect of training package assessment guidelines on that choice. 

Stage 3 — evaluation and analysis 

A self-assessment tool was developed using the literature review and data from case-study site 
interviews, with national assessment principles as a benchmark. The tool was validated in 
consultation with the project reference group and a technical expert. It allows providers to self- 
assess practice in the areas of validity and reliability of the assessment methods and processes. 

Stage 4 — the report 

The final stage of the project involved confirmation of case-study details with respondents, writing 
the report and refinement of resources for providers. 



Limitations in the study 

Methodology 

Interviews of respondents from the case-study sites were conducted by telephone, recorded with 
permission, and then transcribed. These interviews generally ranged from one to one-and-a-half 
hours. Supporting documentation was also collected where available (including course information, 
information about assessment and examples of assessment tools). 
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While the project brief specified interviews with a number of personnel from each of the providers, 
this was an ambitious goal and proved neither possible nor feasible in most cases. 

In terms of accessing supporting documentation, some registered training organisations had 
purchased learning guides with accompanying assessment tasks not available to the researchers. In 
other cases the training provider had course design teams and product development teams 
physically removed from the delivery site, or the original personnel were no longer available to be 
interviewed. 

In the context of further research in this area, it may be useful to consider further investigation of 
the assessment ‘chain’ in terms of the roles, responsibilities and issues and impact for personnel 
involved at all stages of the assessment process. 

Defining flexible delivery modes 

Flexible delivery and learning arrangements are evolving concepts in the VET sector, carrying a 
range of meanings for different people. 

At the outset of the project a framework was sought to enable a categorisation of the current 
models of flexible VET delivery. There was a deliberate emphasis on models not using significant, 
formal face-to-face, classroom-based instruction. This parameter was set because of the greater 
interest in the issues posed by the lack of regular face-to-face contact with students. 

While the provider matrix (appendix 2) assigns a particular delivery mode for each case-study site, 
the framework was not always easily applied. In appreciation of the potential diversity of 
arrangements for flexible delivery, the survey allowed for multiple responses and it was rare for any 
respondent to nominate one mode only. ‘Mixed mode’ was the most frequendy nominated mode. 
Thus, for the purposes of the study, in the provider matrix the dominant element of the delivery 
arrangement for the module or course has been used to assign it to a particular category. In reality, 
a large proportion of flexible delivery models at this stage are mixed mode or ‘hybrid’, often 
incorporating some component of face-to-face activity (such as residential attendance, orientation 
programs or tutor visits to a work site). 

While a conceptual framework relating to technologies used in flexible delivery (the technology 
generation framework, noted in the literature review) may be one way to differentiate between 
delivery types, the terms selected for the project are those most commonly used by VET 
practitioners — such as distance, centre based and online. Broadly speaking, case-study sites were 
identified using the following principles: 

V Distance delivery is identified as a mode in which primarily print-based learning guides and 
other resources are sent to learners, and the majority of study is undertaken at home or work. 

V Online delivery is used when learning and assessment resources are loaded onto websites. 
Learners log in from home, work or another location. 

V A learning centre in a college or workplace primarily facilitates a self-paced and resource-based 
learning approach. Teachers or trainers are available in the centre to provide support, arrange 
tutorials and assessments. 

V Mixed mode uses a combination of delivery arrangements (distance, online, face-to-face, 
workplace-based etc.). 



Provider survey 

Respondents were not always familiar with the national training package qualification or 
competency codes and sometimes used local module descriptors and codes. In two cases it was 
difficult to assess whether the course was a national qualification because of the lack of initial 
information about the course code. 
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Self-assessment of practice 

Limitations in the final part of the third stage of the study — self-assessment of practice — include: 

-y- the contribution of a small data set to further insights about validity and reliability in VET 
assessment practice in flexible delivery 

-y- the nature of the sample for this study precluding the researchers from making definitive 
statements about the validity and reliability of the assessment methods 

-y- the usefulness of yes/no responses to this type of data 

self-assessment or self-reporting as a more appropriate approach to the evaluation at this stage 
because of the sensitivity and confidence levels 

comparisons and conclusions that can be made from the responses provided by the 
respondents. 
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What the literature says 



Introduction 

The literature review is the first stage of the study. 

The review covered: 

4- national VET policy documents and research in assessment and flexible delivery 
<■ VET assessment practice in general 

4- VET assessment practice within a flexible delivery framework 

4- theoretical views on assessment in a standards-driven, competency-based system 

4- experience in other educational or related sectors that might be relevant to the Australian VET 
context. 



Broader context for flexible assessment in flexible delivery 

The reasons for the adoption of a competency-based approach to vocational education and training 
in Australia and the history of its implementation are well documented. The key components of the 
reformed VET system — an open training market, industry input, user choice and cost-effectiveness 
(ANTA 1 993) — also underlie current challenges and issues relating to assessment in a competency- 
based, flexibly delivered VET system. Such challenges include not only the implementation of 
training packages and the Australian Quality Training Framework, but also meet the need for 
consistency to underpin confidence in the national mutual recognition principle (ANTA 1999). 
Further, there is the growth in interest in technology-enhanced learning environments. The 
potential of current and emerging communication and information technologies to support and 
manage both assessment and learning processes is clearly a significant issue for VET providers and 
policy-makers. 

As training delivery and learning arrangements in the sector adapt to meet changing skill needs in 
the community and economy, and new technologies, it is important that the role of assessment and 
methods of assessment are revisited. 

While valid, reliable, fair and flexible assessment is the cornerstone of the Australian VET system 
(National Assessment Principle 8, ANTA 1999), it is possible that, in a competency-based system, 
and with an increased focus on new technology use, there are tensions between the qualities of 
reliability and validity on the one hand, and flexibility on the other. These need to be explored. 

Flexible delivery is a key priority. Strategic planning and directions for this are provided through 
the previously mentioned Australian Flexible Learning Framework 2000—2004. Foremost among 
challenges to practitioners in vocational education and training is the need for assessment models 
that can achieve in their design, a balance between the needs of the individual, the needs of 
industry and the requirements of a national system. 
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A considerable amount of research has reported on the flexible delivery of VET programs (Kearns 
1997; ANTA 1997a, 1997b, 1997c; Misko 1994; Booker 2000; Trood & Gale 2001). Yet there 
appears to be little research to date that has quantified or qualified the range and rationale for 
approaches to assessment in these programs (Thomson, Saunders & Foyster 2001). Reasons for 
the selection and use of assessment methods by practitioners have been largely overlooked 
(Docking 1995). 



Defining flexible assessment in vocational education 
and training 

The term ‘flexibility’ is used in a range of social, economic, political and educational contexts. In 
recent years it has increasingly been associated with responsive strategies used to adapt to 
constantly changing conditions and environments as well as to customer or client needs. 

The concept of flexibility in assessment in the Australian VET context has its origins in the 
National Framework for the Recognition of Training (1992) and Vocational Education, 
Employment and Training Advisory Committee documents. The former outlines key principles of 
assessment, with the already stated view that ‘sound assessment in a competency-based system has 
four essential features — it is valid, reliable, fair and flexible’. Furthermore, ‘the method of 
assessment may be flexible, but must measure what it says it measures, be reliable, valid’ (Hager, 
Athanasou & Gonczi 1994). The assessment principles are further elaborated in the Vocational 
Education, Employment and Training Advisory Committee (1993) document, Framework for the 
implementation of a competency-based vocational education and training system-. 

The assessment practices endorsed for the implementation of CBT [competency-based 
training] must be flexible if they are to be appropriate to the range of knowledge and skills 
encompassed by CBT and if they are to be appropriate to the range of delivery modes, sites of 
delivery and the needs of learners. There is no single approach or set of approaches to the 
assessment of performance in a competency based system. 

(Vocational Education, Employment and Training Advisory Committee 1993) 

Hager, Athanasou and Gonczi describe flexibility principles in assessment practice as those in 
which: 

Assessment should cover both the on the job and off the job components of training. 

Assessment procedures should provide for the recognition of competencies no matter how, 
where or when they have been acquired. 

Assessment procedures should be made accessible to learners so that they can proceed readily 
from one competency standard to another. 

(Hager, Athanasou & Gonczi 1994, pp.139-40) 

The Training Package for Assessment and Workplace Training (BSZ98) reiterates those statements 
and qualifies the application of flexibility in assessment, by noting that: ‘Flexibility applies to the 
process — not the standard. Adjusting the standard beyond “reasonable adjustment” can affect the 
validity of the assessment’ (ANTA 1998, p.19). 

Accommodating the scope, context and range of needs involved in assessment is also emphasised 
by Rumsey: 

Assessment practices are flexible if they can accommodate the scope of knowledge and skills 
encompassed by the assessment criteria, the variations in contexts in which assessment may 
be conducted and the range of needs and personal situations of potential candidates. 

(Rumsey 1994, p.20) 

This is restated by Misko (1994, p.24) when she notes: ‘varying assessment to fit in with changes in 
delivery gives integrity to the whole process’. 
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While validity, reliability and fairness are also well-developed concepts in educational measurement 
and testing, flexibility is generally seen as a practical consideration, derived from a notion of 
‘adaptivity’. It is possible in fact that the term ‘flexible assessment’ became part of the vocabulary 
of the Australian VET system as a consequence of adopting more flexible approaches to delivery 
and recognition of skills, particularly on-the-job training and delivery. Earliest references to the 
term in VET literature seem to emphasise the ‘intention’ of the principle of flexibility, which was to 
encourage assessment practice that would be appropriate and adaptable to the range of delivery 
modes, sites of delivery and the needs of learners. 

Under flexible assessment principles, particular reference is then made to: 

V the coverage of both on- and off-the-job components 

V recognition of competencies regardless of how, where, or when 

V accessibility of assessment procedures to learners. 

(Hager, Athanasou & Gonczi 1994, p.15) 

Early discussions of the concept of flexibility in Australian VET assessment often refer to varying 
the ‘when’, ‘where’, ‘how’ and by ‘whom’ of assessment being carried out (Misko 1994, p.24; 

Hager, Athanasou & Gonczi 1994, p.15). Hager, Athanasou & Gonczi (1994), for example suggest 
that: 

Another way in which the assessment techniques can have flexibility built into them is by their 
making use of some of the newer and less formal types of assessment, such as portfolios or 
peer assessment, which by their nature offer greater flexibility. 

(Hager, Athanasou & Gonczi 1994, p.15) 

Variability in timing, location and method has generally been accepted as a basic feature of flexible 
assessment. However, the notion of flexibility underpins both a process of assessment and an event of 
assessment. Flexibility needs to be built into an approach to assessment, guided by the context and 
purpose of the assessment. It has to be demonstrated in the assessment strategies used by the 
registered training organisation. As the literature shows, however, the term ‘flexibility’ is used in 
relation to a number of aspects of assessment — flexible assessment methods, techniques and 
approaches. 



Tensions between validity, reliability and flexibility 

The literature has little to say about the potential challenges for assessment design and processes in 
a flexibly delivered VET program, in regard to a potential tension between validity, reliability and 
flexibility. However, some reference is made to the possible tensions between consistency and 
flexibility (Toop, Gibb & Worsnop 1994). 

Smith (2000) suggests that flexibility in assessment in the VET sector has not been realised, except 
perhaps in regard to locations and timing. To restate and further that argument here, he asserts that 
while: 

. . . the last decade has seen the emergence of a new and significantly refocused paradigm for 
training, this has not been accompanied by a new paradigm for assessment . . . the system has 
generally attempted to fit existing assessment methods to the ‘new’ training environment, 
rather than taking a ‘paradigm zero’ approach which starts with the characteristics of the new 
training environment and develops the most appropriate and cost effective ways of 
confirming the attainment of outcomes in that environment. (Smith 2000, p.25) 

Thus, there is a concern that over-reliance on written assessment tends to place a stronger focus on 
the assessment of underpinning knowledge, rather than the demonstration of a skill or the 
application of the knowledge to the skill. It needs to be asked then, if distance programs in a 
competency-based system can rely on one method of evidence collection for valid judgements of 
competency. 
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In summary, the current understanding of the term ‘flexible assessment’ relates to assessment 
practices that demonstrate an understanding of the learner’s context and needs in relation to 
assessment, while at the same time maintaining validity and reliability in the assessment process 
(Reynolds 1997). The latter qualities are discussed below in relation to competency-based 
assessment. 

Other contexts for flexibility in assessment 

An interest in flexibility in assessment is, of course, not confined to the VET sector. Other 
education and employment-related sectors are also reviewing practice in relation to assessment, 
particularly how technologies might be harnessed to support both education and employment- 
related assessment services. 

There is considerable experimentation with technology-enhanced assessment in the tertiary sector. 
Literature relating to assessment approaches and methods in the tertiary sector is referred to more 
fully later in this chapter under the heading, ‘Assessment in flexible learning — the tertiary sector’. 

In this section key issues which are currently or about to confront the VET sector are highlighted. 

Human relations, recruitment and skills recognition 

Another area of relevance for the VET sector is the development of ‘flexible’ assessment in the 
human resource and recruitment sector (< www.hr -guide. html>). This area has traditionally used a 
range of research-based psychometric testing processes to: 

-y- assess vocational competencies and job readiness 

-<> assist selection and recruitment activities 

-v- assess a range of other skill and attitudinal behaviours. 

The ability to demonstrate or provide evidence of key workplace competencies is a high priority for 
prospective employers. 

Recognition of prior learning or current competence are aspects of assessment in the Australian 
VET sector and acknowledged as an appropriate approach for the delivery of full or part 
qualifications to existing workers. Currently, a range of registered training organisations is 
experimenting with online delivery of this assessment, although much of the activity so far is 
related to providing information and initial applications. 

Other organisations are drawing on models from the human resource sector to support 
recruitment activities — developing online services to link the candidate to an initial self-assessment 
against the training package competency standards, and / or for the identification of evidence for 
claims for competence. As many of the assessment activities in the human resource areas relate to 
the key and generic competencies, there is a potential interface to be explored in the ‘flexible’ 
assessment arrangements for the VET and human resource sectors. The growth of these services 
by human resource organisations can be seen in the number of websites promoting online services 
for these human resource-related assessment activities (for example: <www.westone.wa.gov.au>). 



Educational measurement and testing services 

Large-scale educational testing is a feature of many educational contexts, from primary school to 
tertiary levels. It is used for a range of assessment and reporting purposes. Online assessment 
services are being developed across a number of key assessment agencies both locally and 
internationally (Australian Centre for Educational Research website: <www.acer.edu.au/ 
united/index. html>). In its paper presenting a scenario for the future of large-scale educational 
assessment, the Educational Testing Service (found at <www.ets.org>) provides an insight into the 
future of technology and computer-assisted assessment. 
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Noting the impact of market forces on the testing field, the paper suggests that: ‘The testing 
enterprise has grown dramatically during the past decades, attracting larger corporate presence and 
fuelling competition’. Using a framework of three generations, the paper explores current and 
future possibilities for large-scale assessment that will ‘come to serve both summative and 
formative purposes, be curriculum embedded and performance based, occur at a distance, and 
measure new competencies as the skills valued by society change’. 



Defining flexible learning 

The Australian literature offers a range of descriptions of the term ‘flexible delivery’. For example: 
[Flexible delivery is] a range of approaches to providing education and training, giving learners 
greater choice of when, where and how they learn. Flexible delivery may involve distance 
education, mixed mode delivery, online education, self-paced learning, self-directed 
learning, etc. (Knight & Nestor 2000, p.18) 

Essentially, the concept of flexible delivery brings a range of student- or learner-centred 
considerations into a national VET system. The term is derived from the early national policy 
directions which sought to shift the emphasis from an input model of training, driven in the main 
by the major public training providers, to a customer/industry-based system. ANTA’s National 
Flexible Delivery Taskforce final report describes flexible delivery as: 

... an approach rather than a system or technique; it is based on the skill needs and delivery 
requirements of clients, not the interests of trainers or providers; it gives clients as much 
control as possible over what and when and where and how they learn; it commonly uses the 
delivery methods of distance education and the facilities of technology; it changes the role of 
trainer from a source of knowledge to a manager of learning and a facilitator. . . 

(ANTA 1996, p.8) 

A range of flexible delivery models to support learning continues to evolve in the Australian VET 
environment, and the terms ‘flexible learning’ and ‘flexible delivery’ have often been used 
interchangeably. Researchers have also noted the problems for research and implementation of 
flexible delivery because of the lack of clarity in the flexible delivery terminology (Kearns 1997; 
ANTA 1997a, 1997b, 1997c; Nunan 1996). However, it is now clearer in both the literature and 
national policy directions that the stronger emphasis is on the term ‘flexible learning’. The term 
‘flexible delivery’ is more appropriately applied when it is used in the context of the management, 
planning and systems for learning arrangements. 

Whatever the variations on a theme offered, the defining components or key ‘constructs’ for the 
term ‘flexible delivery’ are generally accepted as the removal of constraints of access related to 
timing of training, mode of training, location of training and learning choices. 

Distance education — conceptual frameworks 

Recent distance education literature makes reference to conceptual frameworks for describing 
distance, open or flexible delivery modes (Bates 1995; Parashar & Philip 1998; Thorpe 1998; Taylor 
2001). These frameworks have shifted the locus of attention from the student’s location as a 
learner, to the characteristics of the technologies used to provide and support the learning 
strategies and resources. The frameworks identify characteristics of what are called first, second, 
third, fourth and, more recently, fifth-generation distance education models (Taylor 2001). 

The features of each generation are determined by the technologies used in delivery and 
assessment, as well as by the communication modes between learners and tutors. 

The first-generation distance education model is characterised by text-based learning resources and 
very occasional written communication between tutor and student. Features of the second- 
generation model (the multi-media model) include integrated audio and visual material with text 
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and improved communication through post, fax, face-to-face contact and telephone. This is still a 
strong feature of the Australian ‘distance’ VET system. 

The third-generation distance education model (the ‘tele-learning model’) allows synchronous 
communication via tele-conferences, video conferences, audio/graphic communication, broadcast 
television and radio. 

Much of the recent literature focusses on the fourth-generation distance education mode — the 
flexible learning model — in which computing and telecommunication technologies have converged 
to provide online computer-mediated communication. This is used increasingly as part of the 
communication and learning arrangements in distance education (Thorpe 1998). 

While there are general assumptions that successive generations of distance education mark 
changes for improvement in the learning arrangements, Thorpe (1998, p.266) suggests that there 
are still significant issues in relation to the amount and quality of student participation in electronic 
learning environments. 

Experience in distance education has confirmed that the acclaimed interactivity of third 
generation technologies depends on a context of imaginative and careful design if it is to be 
achieved successfully. (Thorpe 1998, p.272) 

The fifth-generation model — the intelligent flexible learning model (Taylor 2001) — is essentially a 
derivation of the fourth generation that aims to capitalise on the features of the internet and the 
web-building campus, including access to institutional processes and resources. 



Issues for competency-based assessment in vocational education and training 

Under a competency-based assessment system, performance becomes an important source of 
evidence. Concepts of direct and indirect evidence emerge as considerations for assessment design 
in the VET sector. 

The early research into assessment issues for vocational education and training accompanied the 
introduction of a competency-based training system (Bloch & Thomson 1 994; Bloch, Clayton & 
Favero 1995; Hall 1995; Docking 1995). It dealt with a range of issues and implications for 
assessment in competency-based training, including the role and formalisation of workplace-based 
assessment. Bloch and Thomson foreshadow later research interest in the types of validity, 
indicating the then current ‘focus on the unitary nature of validity and the pre-eminence of 
construct validity’ (1994, p.20). They briefly discuss decisions made in assessment processes in 
terms of the support to justify each assessment decision and the decision’s consequences, issues 
that are particularly relevant now under the Australian Quality Training Framework. 

Recent national research into assessment in Australian vocational education and training (Gillis & 
Bateman 1999; Thomson, Saunders & Foyster 2001; Booth et al. 2002; Bateman & Clayton 2002) 
has focussed on issues of quality assurance and, more specifically, on those fundamental 
assessment/ testing issues of validity, reliability and fairness. 

How can assessment be valid, reliable and flexible? 

A substantial literature base underpins the field of testing, assessment and evaluation in educational 
measurement theory in general, and in specific, academic fields of study. Validity, reliability, 
practicality and fairness are considered to be the four key features of good test and assessment 
design. Seminal work by Cronbach (1971), Linn, Baker and Dunbar (1991) and particularly Messick 
(1989) has informed the educational measurement fields over the last three decades. To follow is a 
brief summary of some of the concepts and issues raised, as they apply to this research project. 
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Validity and reliability in the Australian VET system 

Issues of validity and reliability in VET assessment have been the focus of research in Australia 
over the last five years. Important considerations arising from more recent literature relate to the 
changing theoretical perspectives on the nature of validity, and the extent to which VET 
practitioners understand the term and its application. Early references to the definitions of validity, 
for example, appear to have emphasised certain aspects or types of validity and ‘confirmed’ a 
particular understanding of the term in relation to the Australian VET context. Until more recently, 
the focus of validity in assessment has predominantly addressed content validity, such as ‘the extent 
to which the assessment method measures what it is supposed to measure’ (Harris et al. 1995). 

Gillis and Bateman point out that: 

Although there is consensus that all assessments must be reliable and valid, technical terms 
such as ‘reliability’ and ‘validity’ are often ascribed to assessments without sufficient 
supporting evidence, and with a range of different meanings. (Gillis & Bateman 1999, p.8) 

There is an assumption that if industry experts endorse the assessment approach, and if the design 
and assessments are based on the competency standards, the assessments are therefore valid. Gillis 
and Bateman (1999) assert that while meeting these conditions (industry input and matching with 
competency standards) is important, they alone ‘do not provide sufficient evidence to conclude the 
validity of the assessment’. 

More recent Australian research and literature into issues of validity and reliability (Gillis & 
Bateman 1999; Thomson, Saunders & Foyster 2001) have drawn extensively on the theoretical 
perspectives of Cronbach (1971), Linn, Baker and Dunbar (1991) and particularly Messick (1989) 
in building an understanding of these terms. Messick’s seminal paper explains validity as both a 
unitary and multifaceted concept. Chapelle (1999, p.258) provides a table summarising the 
contrasts between past and current conceptions of validity that may help break down some of the 
mystique associated with testing theory and be of use to the VET sector. 



Table 1: Changing conceptions of validity 



Past 



Current 



Validity was considered a characteristic of a test: 
the extent to which a test measures what it is 
supposed to measure. 

Reliability was seen as distinct from and a 
necessary condition for validity. 

Validity was often established through correlations 
of a test with other tests. 

Construct validity was seen as one of three types of 
validity (the three validities were content, criterion- 
related and construct). 

Establishing validity was considered within the 
purview of testing reseachers responsible for 
developing large-scale, high-stake tests. 



Validity is considered an argument concerning test 
interpretation and use: the extent to which test 
interpretations and uses can be justified. 

Reliability can be seen as one type of validity evidence. 

Validity is argued on the basis of a number of types of 
rationale and evidence, including the consequence of 
testing. 

Validity is a unitary concept with construct validity as 
central (content and criterion-related evidence can be 
used as evidence about construct validity). 

Justifying the validity of a test (assessment) is the 
responsibility of all test users. 



Messick (1995) argues that because of the increasing importance and popularity of performance 
assessment and their promised ‘positive consequences for teaching and learning’, it is important 
that the validity of performance assessment be systematically addressed, along with other basic 
measurement issues such as reliability, comparability and fairness. 

Thomson, Saunders and Foyster (2001) also argue for a broader approach to the term ‘validity’ in 
the Australian VET sector, seeing it as ‘the extent to which the interpretation and use of 
assessment results can be supported by evidence’ (p.41). They suggest that ‘it is possible to have a 
wider interpretation of validity that has it subsume principles of reliability, flexibility and fairness’. 
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One of the findings in this study is ‘that assessors clearly needed assistance in getting to grips with 
the greater complexity that comes with the new approach to validity’ (p.39). The point made in 
relation to assessor understanding of technical assessment terminology and assessment principles 
has wider implications for professional development programs in assessment design. 



Building confidence in assessment across diverse VET programs 

The diverse and competitive nature of the VET sector has not always lent itself to a culture of 
openness or willingness to share information about assessment practice. However this climate is 
changing. Case-study material relating to registered training organisation assessment practice is 
growing and more specific data about assessment practices are being shared for research purposes 
(for example, Vocational Education and Assessment Centre 2000; Booth et al. 2002; Bateman & 
Clayton 2002). 

Assessment methods used in the Australian VET sector 

As a general rule, methods should be selected because they are the most direct and relevant to the 
performance being assessed (Gonczi 1993). In other words, there is a match of the assessment 
method with the type of ‘behaviour’ being assessed. 

A general principle underlying the validity of assessments is that the narrower the base of evidence 
for the inference of competence, the less it will be able to be generalised to the performance of 
other tasks. For instance, performance on paper-and-pencil tests in any profession will probably be 
too narrow a base for assessing occupational competence. To avoid the temptation to use a narrow 
and indirect evidence base, a mix of assessment methods for providing evidence to infer 
competence is recommended. 

Hager, Athanasou and Gonczi (1994) identify observation of performance, skill tests, simulation, 
questioning and evidence through portfolio as common assessment methods in competency-based 
assessment. They suggest that these methods may tend to be more easily applied in a face-to-face 
situation. This has implications for the design of assessment in flexible delivery. 



Implementing flexible delivery and learning in the 
Australian VET sector 

A considerable body of policy through the National Flexible Delivery Taskforce (ANTA 1996) 
underpins the development and implementation of a flexible delivery approach to providing 
vocational education and training in Australia. A range of literature also describes and discusses the 
implementation of flexible delivery and learning initiatives (Misko 1994; Kearns 1997; van Staveren, 
Beverly & Bloch 1999; Carroll & McNickle 2000; Booker 2000). It is useful to reiterate some of the 
key points relating to assessment design made by researchers of flexible delivery implementation. 
Issues emerging highlight in particular: 

Sr the support needs of learners in flexible delivery models of learning 
4- the need to assist learners to identify their ‘readiness’ for flexibly delivered learning 
Sr changing roles of the teacher and learner in the flexible delivery model 
Sr ways in which assessment can drive the learning process in flexible delivery. 

Assessment approaches in flexible delivery in the Australian VET sector 

References in the literature to issues relating to implementing competency-based assessment in 
flexible delivery modes are limited. Docking (1995) discusses a number of implications for the 
assessment of learners in remote locations. He highlights a range of issues for stakeholders, 
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including methods used to gather evidence, the separation of evidence collection from judgements 
and, as previously discussed, possible tensions between validity, reliability and flexibility. He 
concludes that what is needed is ‘a concerted research effort to explore the validity and 
effectiveness of assessment options for remote learners and the infrastructure necessary to support 
them’ (Docking 1995, p.133). 

The National Flexible Delivery Taskforce final report (ANTA 1996, p.62) suggests how assessment 
might be made more flexible. Strategies for users include: 

adopting a range of assessment methods, tasks and modes of delivery 

providing users with opportunities for choices among and within assessment methods, tasks 
and modes of delivery 

ensuring assessment methods and tasks are accessible to all potential users 

-y- using assessment as a basis for the recognition of prior learning, leading to variations in 
learning arrangements. 

Examples help illustrate how flexibility can be achieved in practice. For instance Reynolds (1997, 
p.5) identifies the need ‘to think of less traditional forms of assessment for trainees working at a 
distance’ and outlines enhancement such as: 

the use of a workplace portfolio for evidence 

student choice of location of assessment events 

checklists for formative assessment 

-y- the use of media to provide evidence 

-v- refining assessment instruments to address a number of learning outcomes. 

McNickle and Pogliani (1998) report on a small-scale study of the uses of computer-based 
assessment in the Canberra Institute of Technology and four interstate technical and further 
education (TAPE) institutes. While the contexts for each of the programs is different, some 
common themes emerge. Advantages of the assessment methods outlined include: 

the capacity to support student choice of the timing and location of assessment 

responsiveness in terms of feedback 

efficiency in the management of results. 

Specific challenges or concerns identified in these reports are: 
security of online assessment 
assessment of higher order skills 
<■ involvement of a range of others in the assessment process. 

In a project exploring teaching and learning styles which facilitate online learning in the VET 
sector, Jasinski (1998) identifies assessment techniques used in online projects. The most common 
methods are traditional assignments emailed back to the tutor, followed by multiple-choice 
questions marked by the computer, or short-answer questions sent to a database or emailed to the 
tutor for later marking. Only a small number of innovative and integrated assessment techniques 
are cited, such as online debate and presentation of individual papers. 

Webb and Gibson (2000) outline the development and implementation of an online delivered 
Certificate IV Information Technology (PC support) at the Open Training and Education 
Network-Distance Education between 1996 and 2000. The main assessment types used in the 
program were assignments and tests, with no ‘automated’ online assessment. While assessment 
material was online (learners accessed assessment tools on html pages), tests had to be completed 
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within a set time and verified by a supervisor. Two significant issues were identified as constraints 
or challenges for the assessment process in this program: 

V the impact of the Open Training and Education Network’s assessment policies on the 
flexibility of the program’s assessment system 

V the supervision of assessment in the workplace. 

In their consolidation study into the state of Australian online education and training practices, 
Harper and others (2000) suggest that the Australian VET system is in its exploratory and 
experimental stage in terms of online learning and delivery. Their study makes limited reference to 
assessment practice online. They found little use of email submission of assessment items, ‘as many 
subjects required some or all assessment to be conducted in the workplace’. The major assessment 
use of the online environment was via online quizzes (mostly multiple choice, short answer and 
‘drag and drop’) for both formative and summative assessment. 



Assessment in flexible learning — the tertiary 
sector experience 

Research on assessment issues in flexible delivery is more readily available in the higher education 
sector, where there has been a long tradition of distance and open education and, more recendy, a 
rapid take-up of online learning. In fact, there is a sharpening of interest in assessment in general in 
a range of contexts. Online learning has become something of a catalyst for rethinking both 
learning design and the role of assessment in learning. 

Morgan and O’Reilly (1999) discuss the nature of assessment in a distance learning mode and pose 
the question, ‘is assessment a generic issue, or are there unique issues facing teachers and learners 
in open and distance settings?’ (p.20). They suggest that while on the face of it there may be little to 
distinguish assessment processes and practices in face-to-face and distance settings, there is a range 
of issues ‘that impact in subde ways to make the assessment experience for open and distance 
teachers, trainers and learners, different from face-to-face counterparts’. In comparing 
opportunities for assessment in the distance mode and those in a face-to-face mode they assert: 

. . . distance learners are more dependent upon effective, early communication of assessment 
requirements, together with well-designed and cohesive assessment tasks, useful and timely 
support, and a transparent marking scheme that explains how judgements are to be made. 

(Morgan & O’Reilly 1999, p.22) 

The issues raised by Ivendle and Northcote (2000) in their argument for balance in the use of 
quantitative and qualitative online assessment tasks, while based in the context of the tertiary 
sector, are particularly pertinent to evolving online VET delivery models. Their paper proposes 
some of the current challenges facing the online VET sector as: 

V the design of appropriate online learning and assessment processes that actively engage the 
student in their own learning 

V the design and selection of appropriate assessment approaches and methods in an online 
competency-based learning environment. 

They assert that ‘too often educational materials from face-to-face or distance education are 
translated into online courses without any supporting pedagogical transformation’ (p.532). They 
support the view that assessment drives student learning and, as such, ‘assessment should be one 
of the first design considerations when preparing an online course, and be seamlessly integrated 
into the course, not ‘tacked on’ as an afterthought’ (p.532). They concur with Thorpe (1998) in the 
view that: 
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The evolution and adoption of this new online learning context influences the way students 
are being assessed. The impact of such assessment choices are immense and can have far- 
reaching, unintended consequences. (Kendle & Northcote 2000, p.532) 

In their argument for a balanced approach in the use of quantitative (multiple-choice tests, short- 
answer questions, labelling) and qualitative assessment (open-ended, such as portfolios, reflective 
journals, case-based scenarios and collaborative projects) they suggest there has been a 
‘polarisation’ of research into one or the other methods. 

A report of a study of 104 Australian university projects using information technologies in learning 
and teaching for the benefit of student learning (funded by the Committee for the Advancement of 
University Teaching) by Alexander and McKenzie (1998) identifies an extensive range of features 
and factors that contribute to either successful or unsuccessful learning outcomes. While a number 
of the features relates to resources planning and management, others address pedagogical issues, 
including assessment. Among the features identified for unsuccessful projects are that they: 

4- have not prepared students adequately for participation in learning experiences that they have 
not encountered before, such as working groups 

4- have over-estimated students’ willingness to engage in higher level learning activities, especially 
when they are not related to assessment 

4- have not changed the assessment of learning to reflect changed learning outcomes. 

Herrington and Herrington (1998), in investigating the effectiveness of authentic assessment in a 
multimedia learning environment with prospective primary and secondary mathematics teachers, 
apply a set of essential characteristics of authentic assessment to the design of a CD-ROM 
program. The authors develop an assessment process that ‘engages students in applying knowledge 
and skills in the same way they are used in the real world outside school’ (Allenspach et al. quoted 
in Brown 1998, p.21). The authors voice a real concern that, ‘While constructivist approaches to 
teaching and learning are advancing parallel with the utilisation of modern technologies, 
approaches to assessment have failed to keep pace’ (p.319). 

This concern is reiterated by Williams (2000), who argues that a commitment to flexible delivery 
necessarily needs a commitment to flexible assessment. 

Dirks (1998) reports on his research into assessment practice conducted with 21 institutions 
delivering a distance Masters of Business Administration Program. Areas covered include: 

4- advantages and disadvantages of assessment methods 

4- time spent doing assessment-related activities 

4- quality and authenticity of student responses. 

He reports that: 

The professors indicated that they made their assessment choices based on their goals, the 
tool capabilities and the circumstances of the class. Instmctors working in distance learning 
are not creating new forms of assessment and they most frequently choose assessment models 
that they have used or seen used. (Dirks 1998, p.6) 

The paper also raises a number of issues including the relationship between discipline/ subject area 
and the choice of assessment method, time spent on assessment, and the relationship to class size 
and methods of communication for a distance mode. 

Distance and online flexible delivery arrangements pose challenges in using a range of assessment 
methods. A major study of open and distance learning arrangements in higher education 
institutions across 15 member states of the European Union surveyed the assessment methods and 
models used by tutors and institutions (Bartolome & Underwood 1998). One of the major findings 
is that: 
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. . . advances in assessment and evaluation do not appear to have gained favour with open and 
distance education (ODE) tutors . . . While courses may be delivered at a distance, assessment, 
particularly summative assessment, more often than not takes place within the host institution 
or within special authorised centres . . . particularly for courses where some level of 
certification takes place. (Bartolome & Underwood 1998, pp.5,7) 

Thorpe (1998) in her discussion of the importance of assessment, whatever the mode of study, 
suggests that ‘in a distance teaching institution, assessment has if anything an even sharper focus in 
the students’ experience, given the physical distance between students and the accrediting 
institution’ (p.269). Attention to this issue is an important consideration for flexible delivery 
providers. Continuous assessment ‘creates a series of snapshots of the success of the teaching, 
measured in terms of student performance on each assignment’ (p.269). 

Veenendaal (2001) reports on a flexible assessment project for campus and distance learners in an 
online geographic information science program. He identifies the benefits of using online quizzes 
and virtual field trips as being flexibility, early and regular feedback and increased motivation. 
Along with others who are reviewing assessment approaches in the tertiary sector, particularly 
because of the impact of online learning, he suggests that in spite of the lack of change in 
assessment approaches: 

. . . the main issues regarding assessment remain, namely providing timely and prompt 
feedback to students and using assessment to motivate students to keep up to date with their 
study program. (Veenendaal 2001) 



Issues emerging from the literature 

Security and authenticity of student performance 

Natal (1998) and Bartolome and Underwood (1998) refer to the issues of authenticity, verification 
and security of assessment online. Some in the local VET sector see these as important 
considerations in the development of online summative assessment practices (W ebb & Gibson 
2000). A range of strategies is suggested for ensuring security and authenticity (McNickle & 

Pogliani 1998; Morgan & O’Reilly 1999). At least one view is that the concern for cheating is 
exaggerated (Williams 2000) and that in some large-scale, formal face-to-face or take-home 
assessment verification processes are not always in place. 

Assessment of higher order skills and thinking 

The assessment of higher order skills and deep approaches to learning are often cited in discussion 
of the capabilities, advantages and disadvantages of online and computer-assisted assessment 
arrangements. There are diverse views in the literature in regard to the benefits and constraints 
associated with online assessment. 

Much current literature focusses on the extensive use of multiple-choice questions, true/false and 
short-answer formats (Edmonds 1999; Natal 1998; Herrington & Herrington 1998). There is a 
view that the nature of online multiple-choice questions only lends itself to content areas in which 
factual, recall-based and technical information are assessed. Another view suggests that the critical 
issue is really in the design of the multiple-choice assessment; that with careful planning and design, 
multiple-choice questions can be used to assess critical thinking and other higher order skills. At 
the same time there is also a range of experimentation in using tools such as chat rooms and 
bulletin boards for collaborative learning and assessment purposes (Maor 1998; Parashar & Philip 
1998). 
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Assessing teamwork 

Freeman and McKenzie (2000) explore the use of a confidential web-based system for self- and 
peer-assessment of contributions to team tasks in a tertiary course. While the value of group and 
teamwork for the development of learning and workplace skills is acknowledged, its assessment has 
proved a challenge in that ‘Students often enjoy learning in teams and developing teamwork skills, 
but criticise team assessment as unfair if team members are equally rewarded for unequal 
contributions’ (pp.2,18). 



Reconceptualising assessment 

Morgan and O’Reilly (1999) take up the theme of the nature, purpose and function of assessment. 
While their context is mainly the tertiary sector, it is relevant to the earlier point made about 
process / event issues in the VET assessment. They comment on a range of issues and terms in 
assessment but particularly make the point that: ‘We have to reconceptualise assessment as the 
engine that drives and shapes learning rather than an end-of-term event that grades and reports 
performance’ (p.13). Thorpe (1998) also comments on the ‘driving’ nature of assessment in a 
student’s approaches to study in distance education modes. 



Conclusions 

The national and international literature suggests that while assessment is central to competency- 
based education, flexible delivery of VET programs poses challenges for the design of valid, 
reliable, fair and flexible assessment that demonstrates competence against industry standards. The 
relatively few published accounts of a registered training organisation’s experiences with assessment 
in a range of flexible delivery modes means it is not possible to draw conclusions about successful 
approaches to assessment in flexible delivery models. However, the Australian VET examples cited 
earlier suggest that key issues include: 

^ creativity in managing the process of collecting evidence in a non-face-to-face training mode 
using a range of assessment methods 

involvement of learners and others in the collection of evidence 
-v> quality assurance of evidence collection 

-v> balancing the degree of formative and summative assessment 
monitoring and balancing assessment load on learners/ teachers 

integrating learning and assessment to improve motivation and learning using the learner’s 
context in assessment 

developing a range of feedback processes. 
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Provider survey 



Introduction 

This is the second stage of the study. Following an audit of potential participants, providers were 
telephoned to canvas their interest and willingness to participate in the survey. Those who agreed 
were sent the provider survey form (appendix 1), accompanied by a letter outlining project details 
and protocols. The survey was designed to be faxed back on completion. 

Thirty responses were received and are represented here. 

What we asked providers to tell us 

The questions in the pilot fax-back survey request preliminary information in order to identify: 

<■ potential considerations and issues for assessment in flexible delivery 
^ possible case-study sites to meet the requirements of the project brief. 

Key data fields include: 

<■ respondent details 

-v- provider details 

-v- course or qualification details 

-v* learning/ delivery arrangements 

-v* assessment requirements 

assessment design, strategies and methods 
conducting assessment 

-v> general — issues for competency-based assessment in flexible delivery modes and how those 
issues have been addressed 

-v> the respondent’s role 

-v> further contact for the project. 

Industry sectors and Australian Qualifications Framework levels represented 

Table 2 indicates the range of industry sectors and Australian Qualifications Framework levels 
represented in the survey responses. 
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Table 2: Respondent profiles: Australian Qualifications Framework levels and industry sectors 



AQF level Industry sectors or qualifications 

2 Food processing, Business administration, Textile and clothing, Retail operations, Information technology 

3 Education support, Local government, Tourism/Hospitality/Agriculture (Dairy), Boat and shipbuilding, Office 
administration (Indigenous), Drilling, Engineering, Community services (Children's services), Seafood 
(Aquaculture) 

4 Information technology, Business administration, Transport and distribution, Airline operations, Horticulture 

5 Frontline management, Engineering, Quarry management (Extractive industries), Forensic investigation 
(Public safety) 

Other Nursing, Health industry 

Note: AQF - Australian Qualifications Framework 

The information from the survey responses was transferred to a provider matrix that summarises 
the assessment methods used for the nominated courses (appendix 2). 

The sample includes all types of registered training organisations and all states and territories; 
however, the school sector is not included. Views on issues relating to assessment in the flexible 
mode were sought from two New South Wales respondents from the school sector, but it was 
considered inappropriate to include this sector in the study due to state-level considerations that 
impact on assessment practice for final year students. 



Key aspects of the collected data 

Course or qualification details 

background 

The study does not emphasise training package qualifications over accredited courses because of 
the range of possible implementation strategies for training package qualifications, nationally, 
across all registered training organisations. 

The development of training packages and their ‘reviews’ or ‘enhancements’ will be an ongoing 
feature of the National Training Framework. 

Findings 

Twenty-two of the qualifications in the survey are training package qualifications and eight are 
accredited courses (see appendix 2). 

Of the training package qualifications, one is an enterprise training package (of which there are 
currendy eight registered on the National Training Information System website). 

Of the accredited courses, plans are in place for delivery in 2002 of the forensic investigations 
qualification against the training package qualification. Four of the other accredited courses are not 
currendy covered specifically by a training package qualification, although some of the modules 
within these are mapped against training package units of competence. 



Learning/ delivery arrangements 

background 

The survey asks respondents to identify delivery arrangements for the selected course and for the 
nominated modules or units of competency by asking for responses to the following questions: 

<■ What are the learning/ delivery arrangements for the qualification or course you have selected? 
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-v- What are the learning/delivery arrangements for the module/units of competency you have 
selected? 

-v- Where are learners located for their study? 

-v- For the modules/units selected is there a requirement for learners to be currently working in 
the industry? 

-v- Is any assessment conducted in the workplace? 

Findings 

A range of delivery models and learning arrangements emerge from these data. While respondents 
are asked to select from one of four possible arrangements (distance using print and other media; 
wholly online; mixed mode and open or learning centre), many ticked more than one mode and 
this was most frequently combined with the mixed-mode option. 

The majority of responses indicate ‘mixed mode’; however, where specific features of the delivery 
are indicated in the response it is possible to assign a dominant delivery mode as outlined in the 
project brief. 

Assessment requirements and resources 

Background 

Informal discussions with practitioners suggest that in the early training package development 
period, the assessment guidelines were very broad and generic, in particular where the training 
package covered diverse sectors within an ‘umbrella’ industry. Since then there has been a stronger 
emphasis on the development of more specific assessment resources for the training packages. 
More recently, resources have been devoted to training package implementation strategies, many of 
which have been dedicated to the development of guides for assessors in particular industry sectors 
(for example the Rural Industry Training Council and the Community Services Industry Training 
Advisory Body resources). 

The survey seeks to gather basic information about the resources used for the planning and design 
of assessment, in particular: 

Are any of the training package assessment guidelines used for assessment planning for the 
modules/units? 

Are any other assessment guidelines or resources used? 

-v- Are there any special assessment requirements associated with the modules/units and, if so, 
who determines these requirements? 

Findings 

The majority of respondents delivering a training package qualification indicate that the relevant 
training package assessment guidelines are used in their planning. 

Other assessment resources cited include: 

national module descriptors and course syllabus documents 
registered training organisation program guidelines 
T local customised assessment resources 
T ‘off-the-shelf learning and assessment guide resources 
enterprise resources 
workplace assessor resources 
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-<> commercial products 

^ industry training advisory body resources developed for the support of assessment (previously 
non-endorsed components of the training packages) 

research publications 

ANTA National Assessment Principles. 

The major categories of ‘special assessment requirements’ identified by respondents include: 

^ assessment to be conducted in the workplace/ on the job 

access to specific or particular resources for assessment purposes (such as access to crime 
scenes, licensed childcare premises, educational settings) 

<■ supervision of a final assessment (on campus or at a workplace) 

participation in assessment with others (such as in collaborative team-based learning 
arrangements) . 

Generally, these requirements are determined by the training package endorsed components, the 
registered training organisation, or both. Two respondents state that although the modules 
nominated for the survey are not guided by special requirements, other modules in the course may 
be. 

It is interesting to note that the study does not explore in any detail the issue of how registered 
training organisations find out about available assessment resources, or the actual ‘reach’ of these 
resources. This, however, is an important issue and worthy of further investigation. Reference to 
this issue is made in the case-study material. 

Assessment design, strategies and methods 

background — assessment design and strategies 

In flexible delivery models, the diversity of learner needs and contexts for assessment is potentially 
far greater than in the face-to-face model. Flexible delivery, with its implicit focus on a more 
learner-centred approach, suggests a much stronger emphasis on individualisation of assessment. 

The survey seeks preliminary information about approaches to assessment and evidence-gathering, 
particularly the stage at which the approaches to assessment and evidence collection are developed. 
Information about the personnel involved is also sought. The survey asks for responses to the 
following questions: 

At what stage and by whom were the assessment strategies determined for the modules/ units? 

<■ At what point and by whom were assessment methods determined for the module/ units (that 
is, particular techniques or procedures used to gather evidence)? 

These questions are asked: 

to establish the level of involvement of the respondent 
^ to identify the possible levels at which assessment roles and responsibilities are differentiated. 

Findings — assessment design and strategies 

Arrangements for assessment design vary within this small sample of flexible programs. The 
responses in this field of data are quite diverse, and it is clear that individual contexts (such as 
delivery mode and the specific demands of the qualification area) are significant factors. These 
factors are taken up more specifically in the case-study analysis, which provides more information 
about the effects of contexts and conditions on assessment. 
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There is an opportunity here for further research to collect specific data from a larger and 
representative sample, with the aim of providing information about where assessment resources are 
best targetted. 

background- — considerations affecting choice of assessment methods 

Assessment methods are generally understood to be ways in which the five major forms of VET 
assessment (observation, skill test, simulation, evidence of prior learning or achievement and 
questioning) are further broken down into specific ways to collect evidence from these forms of 
assessment (Hager, Athanasou & Gonczi 1994). Competency-based assessment involves the 
collection of a sufficient range and quality of evidence to enable a sound judgement to be made 
about an individual’s competence. 

One of the areas of interest of the study is the determination of factors or considerations that may 
impact on the choice of assessment methods for flexible delivery, and whether these satisfy the 
needs for validity and reliability. This part of the survey asks: 

V What factors or considerations affected the choice of the assessment methods for the 
module/units in flexible delivery mode? 

Findings — considerations affecting choice of assessment methods 

Responses fall broadly into the following six categories: 

V workplace-specific needs and industry sector requirements (that is, work conditions, 
production, schedules, type of work, consultation with enterprise staff, legislative 
requirements) 

V location of the student (distance and remote locations) 

V access to resources to support the assessment (physical resources as well as human resources — 
supervisors and qualified staff) 

V accommodation of individual student needs and preferences (including language, literacy and 
numeracy needs) 

V assessment principles of validity and reliability (including the integration of assessment across 
competencies and the specific nature of the competencies) 

V capacity of computer-assisted assessment and technology to support assessment. 

In the VET field there are many competencies that require practical demonstration and where, 
currently, simulation may not be appropriate, possible or financially feasible. As one respondent 
comments in relation to decisions about assessment methods: 

... it depends on the content, some topics are well suited to remote learning, others are not. 

For example most aquaculture management topics can be done anywhere, but analysis of 
organisms in the digestive tract of a Pacific oyster requires microscopes, dissection kits and 
skilled staff nearby. We can overcome this using microscopes with video transmitters, but it’s 
not productive enough to be viable. 

Three respondents make specific reference to the involvement of the workplace or enterprise client 
and the student in the determination of the assessment strategies and methods. One of the 
respondents (who nominates a combination of the three processes described in the survey) states: 

Both theory material and assessment strategies undergo review and development. While the 
training package is considered in the first instance, this gets refined at design stage, and gets 
tuned and adjusted by teachers at the delivery stage so that the best assessment for online 
delivery can be obtained. 

It is noteworthy that cost-effectiveness issues are not explicidy raised in the survey questions; 
however, further research may investigate the cost issues associated with the different assessment 
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approaches in flexible VET delivery modes. The growth of interest in computerised ‘assessment 
item banks’ is largely attributable to economic considerations in many cases. 

background- — summative and formative assessment methods 

The study seeks to explore the types of evidence used to make judgements, and the ways in which 
progress towards the achievement of the competencies is supported. To do this respondents are 
asked to provide the following: 

V Please describe up to five assessment methods used to collect evidence for achievement of 
competence (for summative or final evidence). 

V What assessment methods are used to support learners and provide evidence of learner 
progress during training (for formative assessment)? 

Formative assessment is generally understood to relate to the strategies used by teachers to provide 
opportunities for feedback or information on progress towards achievement of competence or 
learning outcome. Ideally, these opportunities are designed to allow learners to revise their learning 
needs. These strategies need to be considered in the design process. They may take the shape of 
assessment, involving student interaction with a constructed activity, or be part of an informal 
process of verbal or written feedback, based on observation of performance or contact with the 
student. Good practice in assessment is one that plans and designs the learning experience to 
‘seamlessly’ incorporate the formative model of assessment in such a way that it becomes a part of 
the learning, and not just an ‘assessment event’. 

This philosophical perspective on the role of assessment in learning is not necessarily one that all 
VET practitioners have developed. More recently the focus on flexible delivery and learning and 
the role of new technologies in education have highlighted a need to revisit the nature of learning 
and the theoretical perspectives underpinning learning. 

The differentiation between these types of assessment is made because of the interest in the issue 
of feedback on progress for learners in a flexible mode. The literature suggests that formative 
assessment and feedback are two major challenges for distance and flexible delivery arrangements. 
Morgan and O’Reilly (1999) indicate the importance of assessment feedback, particularly where 
‘assignments may represent the only form of communication between a teacher and a learner’. The 
question is how feedback might be given in distance, open and online delivery modes through tasks 
that are part of the learning process, but that also provide an opportunity for the assessor to make 
judgements about performance. 

While feedback for learners is important regardless of the environment, the literature suggests that 
flexible arrangements need to put more emphasis on this feature, to maintain the learners’ interest 
and motivation. Mechanisms for feedback are considered critical because of the learner’s isolation 
and lack of a supportive ‘learning community’. 

Findings — summative assessment 

Three respondents indicate that ideally the process of assessment is embedded, and that both 
formative and summative assessment should be designed to contribute to the learning process: that 
‘all assessment regardless of whether it is designated as formative or summative is treated as 
formative. All assessment is considered learning experience’. 

Respondents also nominate a wide range of summative assessment methods. Responses have been 
collated using a schema based on the forms of assessment and methods described in Hager, 
Athanasou & Gonczi (1994), where the basic forms of assessment in vocational education and 
training are described as: 

A observation 

V skills test 
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-y- simulation 

evidence of prior learning 
-v> questioning. 

Survey responses have been coded using this schema to gain an overall impression. (The study 
acknowledges the difficulty in making any generalisations based on the responses in this small 
exploratory survey across such differing industry sectors and qualification levels. Comments 
therefore relate to the pilot group only.) 

Examples of summative assessment given by the respondents include: 





computer-based quizzes and exercises 




oral questioning on the 




practical demonstration 




portfolio evidence 




reports (research, investigation, laboratory) 




a practical product 




workplace assessment/ observation on the job 




project 




supervisor testimonial or report — third-party 




self-assessment/ peer 




evidence 




assessment 




oral presentation of information 




case study 




log book or work/job diary 




written test questions. 



Most respondents use more than two forms of evidence as the basis for making an assessment 
decision. Written assessment in the form of questions, assignments or essays appear to dominate. 
The use of assessment guidelines, the learners’ context, the specific nature of the competencies, 
and the demands of the qualification level all appear to have been considerations in the selection of 
assessment methods. 

Findings — -formative assessment 

Most formative assessment activities reported in the survey data include: 

self-check exercises of the multiple-choice question, matching or true/ false types (either in 
printed learning guides or computerised quizzes which give immediate feedback) 

-v> email contact between learners and tutors 

-v> mentoring by learning centre staff or workplace assessor/ trainers 

-y- integrated learning and assessment activities that contribute to the building of skills and 
knowledge in preparation for more formal assessment 

-y- practice assessment events. 

Responses to the manner in which students receive feedback and information about progress 
towards competence fall broadly into six categories. These are described because the explanation in 
the question suggested responses that might indicate both teacher/ assessor roles in feedback and 
information processes, and student activity or responses. An analysis of the responses is shown in 
table 3. 
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Table 3: Formative assessment: Analysis of responses 



Category 


Description 


Number of 
responses 


1 


Teacher/assessor contact through phone/fax/email/chat room/discussion 
board/face-to-face contact 


21 


2 


Self-assessment and reflective activities given in the learning guide activities 
or computer-assisted assessment activities with immediate feedback 


15 


3 


On-the-job feedback by supervisor or assessor 


12 


4 


Completion of activities or practice or 'draft' assessments submitted to 
teacher for comment, advice, marking or contribution to summative 
assessment 


13 


5 


Learner tracking management system 


4 


6 


Use of log book or work diary 


6 



Two respondents indicate that methods used for summative and formative assessment are the 
same, so it is difficult to differentiate in these cases. Of particular interest are those strategies that 
support ‘practice’ events in the assessment process. 

Respondents report on opportunities for students to receive feedback on assessment tasks prior to 
the presentation of final evidence, encouraging them to improve their performance. 

A number of respondents nominate the use of computer-assisted assessment for both formative 
and summative assessment purposes. 

Conducting assessment 

background 

As the role of workplace assessment is a key feature of the national VET system, having access to a 
workplace is a challenge for many providers. Many of the training package guidelines specify 
workplace assessment or ‘simulated’ workplace assessment. In some cases of distance or flexible 
delivery, it may be more difficult for learners to gain access to a workplace while in others they may 
perform the bulk of their learning in a workplace. 

In flexible delivery of vocational education and training, considerations such as the location of the 
student and the capacity to demonstrate evidence of work-based skills may require other assessors 
or supervisory staff to provide evidence. The extent to which training providers draw on resources 
outside their operations is of interest in both flexible delivery modes and the more traditional, 
campus-based face-to-face models of vocational education and training. For the latter, the 
requirement of many of the training packages for workplace or ‘workplace-simulated’ assessment 
means that the classroom-based delivery model has to develop simulation activities to support 
learning and assessment. 

The study gathered basic information on the extent of workplace assessment in the project sample 
by asking: 

-y- Who is involved in making final judgements about the learner’s competence for the 
module/units selected (teacher/trainer/instructor, qualified assessor in the workplace, 
workplace supervisor or manager, other — for example industry expert, peers)? 

Findings 

Twenty of the 30 respondents indicate that some assessment or component of assessment is 
conducted in a workplace. A number of the programs participating in the pilot survey deliver 
traineeships where workplace visits and assessment are mandatory. Among the respondents, the 
majority of partnership arrangements for assessment involve a registered training organisation 
working with an enterprise and the VET delivery is workplace based. 
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The personnel involved in making or contributing to judgements for competency vary. In more 
than 50% of the programs in the survey, a person or persons other than the teacher or trainer is 
involved in the assessment judgement. 

The collection of third-party evidence is recommended in a number of training packages. In some 
instances it would be impossible for the registered training organisation to collect the evidence to 
demonstrate competence without it. Sixteen respondents indicate the collection of evidence for 
assessment purposes that involves personnel and resources external to the organisation. 

Registered training organisations delivering to students in remote areas need to set up 
arrangements with suitably qualified people who can provide information about a student’s 
workplace performance against the competency standards. Where training providers deliver 
specifically to an enterprise client, qualified workplace assessors contribute to the assessment of the 
on-the-job skills. These partnerships in assessment are critical in many workplace-based VET 
programs. 

The introduction of more than one party to the assessment process brings with it both benefits 
and challenges for the organisation and the candidate. These are discussed in more detail in the 
case-study material. 



Issues for competency-based assessment in flexible delivery 

background 

The collection of evidence for the practical, technical and ‘soft’ skills potentially offers challenges 
to some modes of flexible delivery, particularly online and distance delivery. The survey seeks 
practitioner views on competency-based assessment in a flexible delivery mode by asking: 

V What issues are there for competency-based assessment in flexible delivery modes? 

V How have these issues been addressed in the modules/ units you have selected? 

While some of the issues raised by respondents may be applied to competency-based assessment in 
general, a number come into sharper focus or ‘prominence’ in the context of remote and distant 
learners. Many of the responses are broad-ranging and not always specific to the ‘competency- 
based’ aspect of the assessment approach, respondents sometimes using the question to air issues 
of a more general nature. 

Findings 

The most frequendy cited issue relates to authenticity and security in assessment. 

Other issues raised include: 

V the need to design a range of assessment methods to suit a variety of jobs and workplaces 

V the need to provide opportunities for demonstration of competencies when the present job 
does not allow for this evidence to be demonstrated 

V the challenge for flexible and online assessment in distance and online delivery posed by the 
practical nature of some competencies 

V students’ management of time and the submission of assessment tasks on time 

V the opportunity for negotiated and flexible assessment 

V the need upfront for clearly stated learning outcomes and information 

V the need for some aspect of face-to-face time with students, preferably at the beginning of the 
program, to explain assessment processes and support assessment plans for individuals 

V the requirement for different forms of evidence 
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•> the challenge of ensuring that assessment is relevant 

the challenge of assessment of group and team competencies when students are not working 
with others on a module 

4- the need to provide appropriate assessment design that reflects the workplace and caters to the 
delivery medium 

the problem of students struggling with new technologies. 

Some of the strategies developed by providers to address authenticity of student work and 
appropriate assessment design for the learners contexts are presented in table 4. 



Table 4: Issues and strategies given for competency-based assessment in flexible delivery 



Issue 



Suggested strategies 



Authenticity of student 
work or performance 



Providing appropriate 
design to reflect 
workplace practice and 
catering to the online 
medium 



Providing a range of 
assessment methods to 
suit a variety of jobs and 
workplaces 



4 - Significant assessment events are conducted in a supervised environment. 

Arrangements are made for supervision at a college, workplace or other suitable venue 
where a process of verification of authenticity can be established. 

4* More than one method of assessment is used. 

4* Assessment design ensures that the student needs to have completed earlier 

sequential or ‘building’ tasks personally in order to attempt or complete any summative 
assessment. 

4* The use of continuous assessment to develop a profile of the learner’s performance 
levels. 

4* Mixed-mode delivery with a face-to-face component allows teachers to confirm student 
performance or abilities. Assessment can be validated at team meetings. 

4* Regular contact with students allows teachers to build up some awareness of learners’ 
capabilities and support verification. 

Codes of practice in relation to cheating and plagiarism are clearly documented to 
ensure students are aware of the expectations around assessment. Students have to 
sign an official document that they are submitting their own work or the evidence 
presented is in relation to their performance. 

4* Evaluation strategies including seeking learner feedback on the assessment process 
through regular course evaluation. 

4* Professional development activities in place for teachers and assessors. 

Expert instructional design (team of developers) incorporating assessment at the time 
of program development. 

There is a negotiation process with the student around the best assessment method 
for their context. Teacher and candidate jointly identify a range of ways in which the 
competencies may be demonstrated. 

Workplace projects are used to support the relevance of the assessment. 
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Background 

This is a continuation of the third stage of the study, in which 13 of the original 30 sites that 
responded to the provider survey form the basis of case studies. 

Case studies are often seen as valuable preliminaries to major investigations. They offer the 
possibility of rich subjective data and may bring to light, variables, phenomena, processes and 
relationships that deserve more intensive investigation. These case studies demonstrate how a 
variety of registered training organisations has approached some of the challenges of competency- 
based assessment in flexibly delivered VET programs. 

Note, however, that while the case-study sites may be said to provide examples of the range of 
possible models for flexible delivery, they are not necessarily representative. Also, there is no 
intention in the study to identify sites of ‘good practice’, although inevitably examples of good 
practice emerge. 

The case-study data for this project were collected through: 

V one interview 
A researcher observations 

A collection of documents made available by the case-study sites to the researchers. 

A content analysis across the data from the 13 case-study sites draws out the major themes and 
issues. 

Case-study framework 

The interviews seek information on the assessment approaches adopted, including methods, 
information and support available to candidates, and strategies and resources available to assessors. 

Table 5 shows the framework used to organise information for the case-study sites. 
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Table 5: Framework for case-study information 



Aspect 


Information sought 


Course context or 


❖ 


Background to the course or program 


background 


<>■ 


Information about the flexible delivery and learning arrangements 






Profile of learners 






Organisational information 


Assessment 


❖ 


How assessment is planned and designed 


design and 
assessment 




Who is involved in its design 


methods 


❖ 


Factors influencing the choice of assessment methods 


Conducting 


❖ 


How assessment is conducted 


assessment 




Details and arrangements for assessment 






Types of evidence collected 




❖ 


Assessment policies 


Learner needs 


❖ 


How learners find out about assessment 


and support 


❖ 


What information is provided and when is it provided 






Support arrangements in assessment 


Technology in 


❖ 


Ways in which information and communication technologies are used 


learning and 
assessment 


❖ 


Communication issues in flexible delivery/assessment 


Issues for 


❖ 


Key issues for assessment in flexible delivery mode 


assessment in 
flexible delivery 




Strengths of the assessment approach for the course or module 


❖ 


Organisational factors 



Table 6 classifies the sites. The classification is determined by the main ‘delivery’ mode for the 
units or course, as discussed with participants. This classification may be considered arbitrary, as a 
number of the case-study sites can be characterised by more than one of the flexible delivery 
descriptions in the project brief through the development of a mix of learning arrangements. 



Table 6: Classification of case-study sites 



Distance 


Online 


Learning centre 


Mixed 


Certificate III 

Community Services 
(Children Services) 

West Pilbara College of 
TAFE WA 


Certificate III Education 
Support 

Accounting modules 
Box Hill TAFE Victoria 


Diploma of Engineering 
(Computer Systems) 

Torrens Valley Institute of 
TAFE SA 


Certificate II in Food 
Processing Workplace 

West Coast College of 
TAFE WA 


Diploma of Forensic 
Investigation (Bloodstain 
Evidence and Fingerprints) 


Certificate IV in Airline 
Operations (Leadership and 
Management) 


Certificate IV Information 
Technology (Systems 
Analysis) 


Certificate III in Drilling 
(Mineral Explorations) 

ADITC*, NSW 


Canberra Institute of 
Technology, ACT 


QANTAS College, 
Queensland 


Canberra Institute of 
Technology, ACT 




Certificate III in Office 
Admin. (Indigenous) 


Diploma of Extractive 
Industries Management 


Certificate III in Boat 
Building 


Diploma in Frontline 
Management 


Tropical North Queensland 
Institute of TAFE 


lllawarra Institute TAFE 
NSW 


Hunter Institute TAFE NSW 


Pelion Consulting Pty 
Ltd, Tasmania 


Certificate III in Agriculture 
(Dairy) 








TAFE Tasmania 









Note: * Australian Drilling Industry Training Committee 
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Major themes and issues 

Course context or background 

This small sample of case-study sites reflects the diversity and complexity of the VET sector across 
Australia in terms of size, location, scope of delivery and learner groups. The study reveals how 
difficult it is to describe activities in the VET sector as if it were one uniform sector. 

Some case-study sites in this study have characteristics in common. For example, some have a 
strong emphasis on a delivery and assessment relationship with an enterprise or enterprises; other 
sites, for the purposes of assessment and delivery, have a more direct relationship with the learners. 

The case-study sites cover a variety of VET contexts: 

T registered training organisations working directly with industry and enterprise clients 

T registered training organisations working directly with individuals (both in distance and 
learning centre operations) 

T enterprise registered training organisations, providing training to their own employees. 

Learners in the sample group cover the full spectrum of the VET client base, from traineeship to 
managerial levels. The sample also covers learner groups in varying degrees of remoteness from 
their organisation, tutor or teacher. 1 

Assessment context 

The assessment contexts — that is, ‘the environments in which the assessment is carried out’ — vary 
across the case-study sites. The Training Package for Assessment and Workplace Training notes 
that: 

Physical and operational factors, the assessment system within which assessment is carried 
out, opportunities for gathering evidence in a number of situations, the purpose of the 
assessment, who carries out the assessment and the period of time during which it takes place. 

(ANTA 1998) 



Assessment design and methods 

How assessment is planned and designed 

Role of the training package 

All informants indicate that attention to the training package competency standards, the 
performance criteria, the rules of evidence, and the learning outcomes in course curriculum 
documents is the important starting point for any assessment planning and design process. Overall, 
respondents feel that training package assessment guidelines provide very broad guidelines, 
requirements and statements about the design and conduct of assessment. As one commented: 
‘There is not a lot of guidance around assessment in the training package. They need to be generic 
because of the diversity of contexts for the assessment’. 



1 The terms tutor, teacher, lecturer etc. appear in the case studies as appropriate to the particular case-study 
site. 
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The real work in assessment design is at the implementation stage, with the challenge for some 
case-study sites being the requirement in many training packages for assessment in a workplace or 
simulated workplace environment. 

Increasing diversity of assessment contexts 

In terms of assessment design and planning, there are different processes for different contexts. 
The study highlights the increasing diversity of assessment contexts as the national training system 
evolves. Not all the informants are involved in all stages of assessment planning or design. A 
registered training organisation, for example, may purchase learning and assessment packages in 
which a large amount of the assessment activity has already been developed. 

Assessment design and planning models 

Some of the assessment design and planning models include: 

<■ a collaborative design process involving the registered training organisation and specific 

industry sectors or enterprise input, for example the Illawarra Institute TAFE NSW with the 
Mining Industry Training Advisory Body and Quarry Institute of Australia and West Coast 
College of TAFE WA with a food processing enterprise and the Australian Drilling Industry 
Training Committee with drilling enterprises 

^ a registered training organisation team-based approach involving subject matter experts, 
instructional designers and expert users at Canberra Institute of Technology, QANTAS 
College, Torrens Valley Institute of TAFE SA and Box Hill TAFE Victoria 

^ customisation for specific enterprise needs at TAFE Tasmania, Pelion Consulting Pty Ltd and 
the Australian Drilling Industry Training Committee 

-v> sourcing of available learning and assessment resources and adapting these for local use as 
required at Tropical North Queensland Institute of TAFE and West Pilbara College of TAFE 
WA. 

Who is involved in its design ? 

Industry and enterprise partnerships 

The spirit of forging partnerships between industry and registered training organisations is one of 
ANTA’s goals. A feature of a number of the case-study sites is the involvement of a training 
provider with specific industry bodies or enterprises that play a role in the various stages of 
assessment processes. 

Examples include: 

-v> For the Illawarra Institute of TAFE NSW, the design of assessment for the Diploma in 
Extractive Industries Management involves significant industry influence and input. The 
process involves the Institute of Quarrying, the registered training organisation, the industry 
training advisory body, the Primary Industries and Natural Resources Division of TAFE NSW 
and industry consultants. 

The Australian Drilling Industry Training Committee works closely with a range of small-to- 
medium-sized drilling outfits to deliver training and assessment services. The nature of the 
industry and the needs of the enterprises have largely shaped the assessment approach 
emphasising on-the-job assessment. 

-y- In delivering qualifications from the Food Processing Training Package, the West Coast 
College of TAFE WA negotiated the assessment approach at the beginning of the program 
with the client and then developed it in conjunction with the enterprise. The assessment 
approach has been designed to fit with the needs of the enterprise. 
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-v- TAFE Tasmania delivers an Agriculture (Dairy) traineeship program. A key feature of the 

approach to assessment for this industry sector is the context of the enterprise. The registered 
training organisation must be able to ensure that competencies are achieved in relation to both 
enterprise and industry standards. 

V Pelion Consulting Pty Ltd delivers the Diploma of Frontline Management through a 
recognition process. While assessment is informed by the competency standards, an important 
consideration in the delivery of this qualification is the involvement of workplace managers 
and sponsors. Organisational involvement is thought to be a critical component, particularly 
where the current nature of the candidate’s work may not allow demonstration of all 
competencies. 

Features of the assessment process highlighted in these examples include: 

-v- assessment design as a partnership or collaborative arrangement 

V assessment that must be contextualised to the workplace 

V assessment arrangements that are negotiated with the sponsor (employer) and the candidate 
due to workplace demands 

V the registered training organisation having responsibility for ensuring the assessment principles 
are applied 

V a greater likelihood of qualified on-site workplace assessors or trainers involved in the 
assessment process and auspicing arrangements that must be quality assured. 

In the context of enterprise or industry-based assessment, two respondents comment on the scope 
of the ‘ownership’ of, and commitment to, the assessment process on the part of the enterprise. 
This changes the role of a mainstream VET provider, as one respondent says: ‘they really owned it 
. . . we played a consultancy role’. In this context the assessment process is linked to organisational 
requirements and goals and assessment methods need to be devised, documented and agreed to by 
the organisation. A range of assessment methods is described for these particular programs, 
including observation of performance and portfolio evidence from the workplace. 

These factors are important as they suggest a change in thinking about who is involved in 
assessment practice and in the provision of resources for assessment. The increasingly 
‘individualised’ assessment process associated with flexible delivery (in order to meet the needs and 
expectations of both enterprise clients and individual students) will need to be acknowledged in the 
changing roles of VET practitioners. 

building in assessment from the outset 

Four of the programs discussed in the case studies have been designed specifically for a flexible 
learning mode online, learning centre based, or a mix. While they draw on earlier traditional models 
for resources, essentially they start from scratch. In these cases assessment design tends to involve 
teams of practitioners who have a stake in the program. They typically look more closely at a 
learner-centred approach or an enterprise/industry perspective in assessment. 

Examples include: 

V Torrens Valley Institute of TAFE SA, Diploma of Engineering, Computer Systems 

V Canberra Institute of Technology, Certificate IV in Information Technology, Systems Analysis 

V Box Hill TAFE Victoria, accounting modules 

V Hunter Institute TAFE NSW, boat building 

Developing a new learning program offers an ideal opportunity to re-examine assessment in light 
of the delivery mode, the training package requirements and the learners’ contexts and needs. 
‘Greenfield’ sites often adopt an approach that allows for a range of input to the assessment 
planning and design stage. 
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Because of our preoccupations with shaping effective and creative learning experiences for 
our students we tend to overlook the importance of assessment as the rather more powerful 
influence on student learning. (Morgan & Reilly 1999, p.21) 

Purchased learning and assessment packages 

Assessment for the courses described at two of the case-study sites is done through the purchase 
of ‘learning guides and assessment packages’. These provide the underpinning knowledge and 
theory to support the delivery; formative and summative assessment tasks and tools have been 
built into the resources. These resources have generally been designed for the distance learning 
market and have traditionally used written assignments for assessment. 

In this context the local teacher or assessor in not necessarily involved in the curriculum 
documentation or assessment specifications; however, the extent to which these guides and 
assessments are related to specific environments is left to the provider and the assessor. 

Examples include: 

West Pilbara College of TAFE WA, children’s services 

-v- Tropical North Queensland Institute of TAFE, Faculty of Aboriginal and Torres Strait 
Islander Studies Centre, Community Management Program. This program delivers 
qualifications from the Business Administration Training Package to Indigenous students in 
local and remote regions in Northern Australia. It is particularly keen to ensure that the 
resources are culturally appropriate, and that assessment methods take into account the 
learners’ needs and contexts. To do this, Tropical North Queensland Institute of TAFE has 
developed supplementary assessment material where necessary. Assessment of workplace skills 
(particularly for trainees) is conducted either at a candidate’s workplace, or during the 
residential program conducted for many of the modules in the program. 

¥ actors influencing the choice of assessment methods 

Assessment methods 

A range of assessment methods for the collection of different types of evidence is identified (see 
the case-study summaries and appendix 2). 

In all cases the methods selected allow the collection of evidence necessary to make judgements 
about competency. In both the online and self-paced printed learning guides, self-check formative 
assessment is used extensively. Online summative assessment is also used in the online case studies. 
The considerations leading to the choice/ selection of methods are numerous. 

Those mentioned include: 





training package requirements 




experience and skill level of candidate 




competency standards 




the way work is done in industry sector 




enterprise needs /industry needs 




the type of competency to be assessed 




specific learner needs 




genuine or authentic assessment situations 




location of the learner 




integrating learning and assessment 




learner styles 




flexibility but integrity 




multi-faceted nature of qualification 




capacity of computer- assisted assessment 




qualification level 




holistic approach to assessment. 
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Authenticity 

In a competency-based assessment system a number of challenges exist for providers to ensure 
that the evidence collected is both valid and reliable. In the absence of an authentic workplace in 
which skills can be demonstrated and observed, some flexible delivery providers have created 
opportunities to simulate workplace skills or activities so that assessment can be as authentic as 
possible. The principle of making the assessment authentic underpins the learning and assessment 
practices of a number of the case-study sites. 

The following three examples illustrate this principle: 

-y- Canberra Institute of Technology, Diploma of Forensic Investigations: a mock crime scene is 
‘simulated’, during the residential block, for the Bloodstain evidence unit. In this way the 
practical component of the module is assessed through practical tests and an analysis of the 
mock crime scene. Laboratory and investigation reports assess the practical components of the 
Forensic physics unit and a case study is used to assess the learner’s ability to apply knowledge 
to a ‘real’ situation. As a case-study informant explained: ‘When you enrol in something you 
want to relate to what you are doing . . . and so we try to target all the assessment to the police 
context ... we find if we use police jargon and relate what they will do in the real world, we’ll 
get a much better response’. 

4- Torrens Valley Institute of TAFE SA, Diploma in Engineering, Computer Systems: with the 
introduction of flexible delivery for this course, assessment has been designed to reflect, as 
much as possible, the workplace demands of the industry sector. Practical competencies 
require the demonstration of hand skills in soldering, assessed through observation in the 
learning centre laboratory. The integrated assessment of the key competencies with technical 
and theoretical skills and knowledge is a major feature of assessment. Involvement with 
industry has confirmed the view of the faculty that: ‘employers value the key competency skill 
areas as much as they do the practical and technical skill areas’, and that ‘assessment has to be 
flexible to fit in with the flexible delivery’. 

Canberra Institute of Technology, Certificate IV Information Technology, Systems Analysis: 
using an ANTA Toolbox for learning resources and assessment, a virtual marketplace has been 
developed, using online resources and a project-based approach in order to assess the team- 
based methodology and skills required for this industry sector. The approach to assessment 
reflects the team-based structure of the learning environment. 

Typically, these programs have had the advantage of building the learning and assessment 
specifically for the flexible delivery model. 

Third-party evidence 

The use of third-party evidence for assessment purposes is a cost-effective feature of flexible 
assessment arrangements. Often this approach is recommended in the training package assessment 
guidelines to ensure that a variety of evidence is collected over a period of time. In many cases it is 
also used out of necessity. 

A number of factors may influence the use of this method, such as: 

tc the location of the student in relation to the registered training organisation 

-y- the nature of the competency is such that it is both costly and impractical to observe 
candidates direcdy. 

A number of programs offered at the case-study sites use information provided by workplace 
supervisors, managers or workplace assessors to inform assessment judgements made by the 
registered training organisation. 
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Examples include: 

4 - West Pilbara College of TAFE, WA, Certificate III in Children’s Services: candidates are 

required to be assessed in licensed childcare premises. Reports from supervisors are part of the 
evidence considered by the teacher of the program to determine competence. 

TAFE Tasmania, Natural Resources, Certificate III Agriculture (Dairy): to meet the 
competency standards of the Milk harvesting and Care of calves units, candidates need to be 
employed on farms where these resources and livestock are available. Candidates have to 
demonstrate competence in the workplace. As it is not always possible nor economically 
feasible for a teacher to assess all the skills and knowledge components associated with these 
units over the time required, the reports or testimonials provided by the employer or 
supervisor are part of the necessary evidence if it is to be valid and reliable. 

Pelion Consulting Pty Ltd, Diploma in Frontline Management: satisfying the requirements for 
this course often requires the presentation of a portfolio of evidence. Many of the 
competencies within this diploma involve process-oriented activities on the part of the 
candidate. To support achievement of competence, evidence may need to be collected from a 
range of sources within a workplace, including testimonials or employer performance reports. 

Issues raised by informants in discussing this feature of assessment include: 
access to adequately trained workplace personnel to provide the evidence 
quality-assurance arrangements to ensure reliability in assessment 
resourcing of third-party evidence arrangements. 

Two informants also raised the issue of whether workplaces (particularly smaller enterprises) have 
the expertise, the willingness, or the resources to support workplace assessment. The involvement 
of third parties in some assessment processes highlights issues around roles and responsibilities in 
relation to assessment, the need for educating workplace personnel and for quality-control 
procedures. 

The potential for sources of error in performance assessment and strategies to increase reliability 
are discussed by Gillis and Bateman (1999). They propose sources of error as: 

-y- characteristics of the assessor (for example, preconceived expectations of the competency level 
of the candidate) 

^ context of the assessment (for example, location) 
range and complexity of the task(s) (p.28). 

Tensions between flexibility in assessment arrangements and validity and reliability are likely to be 
exacerbated when clear and agreed quality procedures for all parties involved in assessment 
judgements or evidence collection are not fully implemented. 

On-the-job performance assessment requires the consideration of a broad range of factors to 
ensure reliability and validity. As one case-study informant commented, ‘One very important issue 
for assessment in the workplace is the physical needs of the candidate, for example the timing of 
the assessment is important. It is not good to conduct assessments at the end of a shift’. 



Conducting assessment 

How do learners in flexible delivery arrangements provide the evidence required to achieve 
competence? Are there more opportunities or constraints for assessment in flexible learning 
arrangements? What kinds of arrangements have organisations developed to meet the assessment 
needs of learners in these contexts? 

The case-study summaries outline how assessment was conducted for each of the programs. No 
attempt to compare the methods is made given the diversity of the programs and their contexts. 
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However, of great interest for this area of the study is the flexibility of assessment in terms of its 
location, timing and the methods used for summative assessment. Many of the factors relating to 
this aspect of the research are taken up in the section outlining how learners are supported; 
however, some specific examples will give an indication of the approaches taken. 

The degree of ‘flexibility’ around arrangements for assessments varies, and includes student 
negotiation as well as scheduled deadlines. Choices in the method of assessment are available at 
some case-study sites. Students’ locations for summative assessment tasks range from the 
workplace, the training provider campus, an authorised assessment ‘site’ to working from home or 
the workplace at a computer terminal. In all cases, a range of evidence is considered when making 
the final judgement. 

Provisional assessments prior to competence, stating how much more evidence is needed, or the 
number of attempts allowed for the assessment activity, differ across the case-study sites. The 
assessment pathway used by the registered training organisation is determined by the choice of 
methods and the involvement of personnel. Differences relate to organisational policy or 
availability of resources. 

Notable examples include: 

tc Torrens Valley Institute of TAFE SA, Diploma of Engineering (Computer Systems), 

Electronic hand soldering technology module — assessment of theory and practical skills 
Students provide a range of evidence to support their claim for competence, including results 
from computer-mediated and marked multiple-choice questions, short- answer and true/ false 
responses; observation of soldering skills in a workshop using an assessment checklist with 
industry standards performance criteria; research report; demonstration of practical activity and 
use of equipment within occupational health and safety requirements; and inspection by 
student of a soldered product followed by verbal presentation of an analysis of the compliance 
of the product with industry specifications. 

Technology-supported remote assessment is a feature of a pilot program for apprentices some 
hundreds of kilometres from the organisation campus. Using an internet connection, soldering 
boards and web cam equipment, candidates are able to demonstrate the practical skills they 
have developed in as close as possible a simulation of the workplace setting. The timing of 
assessment is negotiated with students, who identify when they are ready for a ‘formal’ 
assessment. Computer-assisted assessment is conducted in a secure room, and bar-coded 
student ID cards are swiped at the time of the assessment event. This information is linked to 
the computerised learning and assessment management system. 

Canberra Institute of Technology, Certificate IV Information Technology, Systems Analysis — 
assessment of team and collaboration skills 

Working online in teams to develop products or processes is a feature of many workplaces. 

The design of learning and assessment for this course is guided by the need to create a project 
team environment, reflecting the real world arrangements for working in the systems analysis 
area of information technology. 

Candidates produce assessment evidence in the form of six ‘deliverables’ from their project as 
‘consultants’ to a virtual retail oudet (developed in the ANTA Toolbox). Of these, two are 
assessed at the individual level and four at the team level. While deadlines are set for the 
production of evidence, continuous feedback allows learners to improve on their performance, 
submitting the ‘culmination’ of their efforts at the end of the period. Candidates need to meet 
the deadlines because of the impact on learning and subsequent completion of the project. In 
the team discussion area learners post their individual and final contributions so that that the 
whole team has access to all the information produced by it. The team submits one product — a 
prototype for the solutions designed by the team. At the presentation, peer-assessment is part 
of the assessment approach. Also submitted are formative assessment activities that have been 
integrated with the learning. 
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The designer of the information technology program identifies a range of challenges for using 
team-based assessment, including the ‘matching’ of people within the team, the composition of 
the teams and cross-cultural issues impacting on communication and team work. 

Assessment issues associated with working in teams, group work and collaboration are extremely 
challenging and pose something of a paradox. Individualism is highly valued in terms of 
remuneration and awarding of credentials and yet work effort is often interdependent and 
collaborative. 

Authenticity of evidence 

Ten informants raise authenticity of evidence submitted by learners as an issue. This is considered 
particularly relevant to online methods and computer-mediated tasks for summative assessment. 
Some practitioners in the distance mode (using mostly written assignment submission by mail or 
email) note a potential for cheating. 

While online technologies offer ‘anytime’, ‘any place’ opportunities for learning, it is not possible to 
be sure that the person at a computer and submitting an assignment is the person enrolled. Some 
informants feel this issue is too difficult to tackle, while others have acted to reduce the risk by 
requiring the candidate to attend a campus or alternative assessment centre where supervision or 
invigilation procedures are organised. 

While strategies such as these reduce flexibility in assessment, proponents argue the advantage of 
greater confidence in the system. Other informants feel that the presentation of evidence on more 
than one or two occasions, and the use of a range of assessment methods, reduces or minimises 
the problem of authentication. 

Learner needs and support 

Support for learners in the assessment process is a critical issue arising from the case-study data, 
and one of the issues most often discussed. As Morgan and O’Reilly (1999) suggest, the ‘windows 
of opportunity’ for clarification about assessment and demonstration of understanding of learning 
are frequently less available in distance and open learning contexts. They caution about the 
importation of an assessment scheme normally used in a face-to-face environment into a distance 
learning environment without due attention to the design, support, communication and 
management of assessment (p.44). It emerges as a common theme across case-study sites. Once 
regular face-to-face contact with a teacher, trainer, mentor or facilitator no longer features in the 
learning and assessment arrangement, a range of support and complementary structures is required. 

The case-study sites employ a rich diversity of strategies to support learners in assessment. Features 
of learner support highlighted in the study include: 





timely and explicit information 


<■ 


mentoring systems 


<r 


established communication channels 


<■ 


learner progress tracking 


<■ 


orientation programs 




choices of assessment methods 


<■ 


tutorial support 


<■ 


assessment plans 


<■ 


feedback processes 




study plans 




formative assessment 




individualised contact 


4- 


practice assessment events 




face-to-face course components. 



Hoip learners find out about assessment 

Providing assessment information at the outset of the program is critical in a flexible delivery 
mode. Learning outcomes and assessment processes need to be stated clearly at the outset and 
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reiterated at appropriate points throughout the program. Assessment information is not limited to 
procedural and administrative arrangements only. A number of the case-study sites explain how 
they have worked through information about recognition of skills processes and the notion of 
‘evidence’ in a competency-based assessment system. 

Some of the methods used to ensure timely and comprehensive information are highlighted in the 
examples below. 

-y- QANTAS College presents assessment information in all student learning resources. This 
information is available online and is also discussed at the mandatory orientation program. All 
assessment activities are documented and explained face to face, or online. 

-y- Canberra Institute of Technology, Certificate IV Information Technology, Systems Analysis, 
presents assessment information at the orientation session and on their website. 

-y- Box Hill Institute of TAFE Victoria, Certificate III in Education (Office Support) Traineeship, 
provides assessment information to learners on enrolment through a personalised email 
contact sheet, as well as in all the study/learning guides. 

^ West Coast College of TAFE WA, Certificate II in Food Processing, provides assessment 
information at the workplace through both informal and formal processes. Briefing sessions 
are conducted before assessment; negotiation around the timing is discussed, and formal 
written notification is given. Support arrangements in communication skills training for a non- 
English speaking background workforce minimise the potentially threatening perception of 
assessment being ‘exams’ only. 

TAFE Tasmania, Natural Resources, Certificate III in Agriculture (Dairy) and the Hunter 
Institute TAFE NSW, Certificate III in Boat and Ship Building provide assessment 
information to learners on enrolment and in the introductory module. All learning guides 
include information about the assessment process. 

-y- Tropical North Queensland Institute of TAFE, Faculty of Aboriginal and Torres Strait 

Islander Studies Centre, Community Management Program provides assessment information 
at enrolment interviews, during orientation, in the introduction booklet and in the learning 
guides for the course. 

Pelion Consulting Pty Ltd, Diploma in Frontline Management, takes account of the 
multifaceted nature of the qualification and the various stakeholders. Assessment information 
involves both ‘sponsors’ (that is, employers) and candidates. Assessment information may be 
given in a group orientation session or at an individual enrolment interview and includes an 
explanation of recognition as a process and the meanings of the terms used: competency, 
evidence and sufficiency of evidence. Practical ways to identify evidence are also explained and 
discussed with participants. 

Illawarra Institute TAFE NSW, Diploma of Extractive Industries Management prepares its 
students for a choice of assessment pathways at the face-to-face orientation program. Learners 
are introduced to the program, to the online learning and assessment resources, the 
terminology of competency-based assessment, the navigation of the online units, and 
requirements for assessment. A self-check activity allows students to assess themselves against 
the competency standards and evidence required for the qualification, while assisting them to 
determine the best assessment pathway. 

The importance of ongoing contact 

Regular communication with students is critical in flexible delivery models to reduce course 
attrition rates, particularly for those with minimal face-to-face contact. Contact with the learners 
whether through telephone, fax, email or post is part of the ‘learner management’ process. There 
are also ‘affective’ issues, such as isolation and confidence, to be considered for many learners in 
distance learning environments. 
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All informants involved in the delivery of an online module or course indicate the importance of 
regular communication with the students. Email contact is often cited as an enhancement to 
communication for many of the sites in the study, on and off campus (the exception being in more 
remote areas where fax is used because of bandwidth and access issues). 

Lack of confidence on the part of the learner, not being sure that they know what they are 
supposed to know, or that they have ‘correctly’ understood the assessment task, are issues that 
need to be taken up by designers and assessors. Developing communication strategies between 
learners and between teachers and learners, has been one of the ways online developers have 
attempted to address peer support and isolation (with varying degrees of success depending on the 
context of the learners). 

Some examples and comments concerning the importance of ongoing contact include: 

A QANTAS College has dedicated resources specifically to this purpose. A co-ordinator regularly 
contacts (emails) students to provide support and identify potential issues for assessment. 

A Two informants discuss the potential detrimental impact of ongoing enrolment on establishing 
effective communication strategies with and between students. In this situation it may be 
difficult to create cohorts of students who are at the same point in the course, and who share 
common learning needs. 

A One informant commented: ‘Learner management is a key feature of the online environment 
infrastructure. There are three program co-ordinators. Their prime role is learner management. 
These are the people who contact the students, keep an eye on where they are at, find out the 
problems’. 

Another informant confirmed this need: ‘You need support in that they go to sleep and go 
dead . . . there might be problems there that you are not addressing . . . you need a facilitator 
who is just contacting people . . . nothing threatening, just ‘How’s it going? Got any problems?’ 

. . . And you have to do it regularly’. 

Support arrangements in assessments 

Issues around flexible timing of assessment 

A number of informants comment on flexibility around the timing of assessment. They recognise 
the need for some structure within a flexible environment, particularly for the assessment of 
candidates. 

For example: 

•4- Box Hill Institute of TAFE Victoria, Certificate III in Education (Office Support) Traineeship, 
formalises the readiness approach through the completion of ‘practice’ assessments. Practice 
assessment is a feature of the program and is mandatory before formal assessment can be 
undertaken. 

While maximising opportunities for flexibility in assessment, some case-study sites identify the 
need to set times for submitting assignments. Other sites have reviewed their earlier ‘flexibility’ in 
regard to submitting assessments because of late submissions and resourcing implications for the 
registered training organisation. As one informant comments: ‘Flexible learning does not mean that 
you can take as long as you like’. 

Analysis of learner evaluation and feedback at one organisation has led to the development of more 
explicit information and guidelines as part of their quality improvement processes. Learner 
feedback has led this organisation to identify that the effectiveness of the tutor’s communication 
with students is a critical issue in learning online. 
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Issues around learner identification of readiness for assessment 

A number of informants speak of structural challenges involved in assigning resources to 
assessment in flexible delivery. In some cases they have reviewed some of the features of flexibility 
in relation to assessment for those reasons. For instance, if the program allows considerable leeway 
to students in relation to the timing of the assessment, this often leads to incomplete assessment 
processes and enrolments carrying over longer periods. 

Experience has also taught that not all learners are capable of self-directed learning and assessment 
arrangements. The development of assessment plans with individuals can ensure an important 
assessment support structure. 

Induction into flexible delivery models needs also to measure the capacity of the student for self- 
directed learning. This is not always explored by registered training organisations, although there is 
a growing awareness that some learners cope well with flexible arrangements and others need a 
great deal of support and ‘scaffolding’, particularly in early stages. 

Current research into support for flexible learning is placing more emphasis on preparation for 
flexible learning and addressing issues of learning styles. As Morgan and O’Reilly (1999, p.66) 
indicate, learners in face-to-face contexts often spend time in class discussing their approaches to 
an assignment or assessment task. Distance learners may often be ‘left to their own devices’. 
Without an effective method for adequately and concisely communicating assessment tasks, ‘their 
effectiveness as a means of promoting learning is lost’. 

The important role of formative assessment 

Ongoing or continuous assessment, designed to support the building of knowledge and skills, is 
more desirable than one or two assessment events. 

The learning centres in this sample operate year round, with ongoing enrolments. As the learning 
approach is resource based, the learners can often choose the module in their preferred sequence. 
Learners identify when they are ready for assessment. As mentioned above, a number of programs 
provided ‘practice’ assessments to support learners. 

Technology in learning and assessment 

In remote and isolated areas, bandwidth and internet service provider problems still limit online 
delivery and assessment; fax and telephone are often preferred. 

Video conferencing has been used in some of the case-study sites (for example Hunter Institute 
TAFE NSW, Torrens Valley Institute of TAFE SA, Tropical North Queensland Institute of 
TAFE). Elsewhere the use of email for the submission of assignments and assessments is 
increasing. 

A range of computer-mediated resources is also being used, mostly for formative assessment. 
Online summative assessment is being trialled and used; normally, however, this occurs under 
supervised conditions. 

Canberra Institute of Technology, QANTAS College, Torrens Valley Institute of TAFE SA, 
lllawarra Institute TAFE NSW, and Box Hill Institute of TAFE Victoria employ short quizzes, 
multiple-choice questions, matching exercises, true/false and short-answer responses as the most 
prominent computer-assisted assessment methods. As the ‘marking’ of these is generally 
automated, students receive immediate feedback. 

If these assessment methods are to genuinely support learning, then the design of the feedback or 
results processes is critical. Well-designed computer-assisted assessment programs tend to link the 
assessment to related learning resources so as to support revision prior to further attempts at the 
assessment. This creates learning loops and helps to minimise the use of chance or guessing. 
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Issues for assessment in flexible delivery 

Other issues that emerged from the case-study interviews include: 
assessing of team skills in a flexible delivery mode 

<■ assessing key competencies 
assessing collaboration 

-y- infrastructure issues for flexible delivery and assessment (including costs for support and 
assessment innovation) 

-y- the role of recognition of prior learning or current competence in assessment approaches. 



Individual case studies 



Case study 1 : Assessing teams in an electronic learning environment 



Site 

Qualification 

Code 


Canberra Institute of Technology 

Certificate IV Information Technology (Systems Analysis) 
IT 40199 


Units of 


TAD 041 A 


Determine client business expectations and needs 


competency/ 

modules 


TAD 042A 


Confirm client business needs 




TAD 043A 


Develop and present a feasibility report 




W027B 


Relate to clients on a business level 




W026B 


Co-ordinate and maintain teams 




TB059A 


Contribute to the development of the detailed technical design 




SP037A 


Contribute to the development of a strategy plan 



Course context or background 

The Department of Software Development in the Faculty of Business and Information 
Technology at Canberra Institute of Technology has diversified its delivery of the Certificate IV 
Information Technology (Systems Analysis) through a mixed-mode, flexible delivery model. The 
mixed mode involves both web-based and face-to-face learning activities. 

Delivery arrangements 

The qualification is delivered through the ANTA Systems Analysis Toolbox, which, along with the 
student guide, is loaded on a web-based educational platform. Students are placed in groups or 
classes and then in teams of four to five. Two flexible ‘as needed’ face-to-face sessions per week 
are scheduled for individuals and teams to discuss and monitor progress. Some formal ‘lecture- 
type’ sessions are conducted during this time and they are delivered as staff development for the 
consultants (students). Students are provided with the electronic resources and access to printed 
texts. Electronic communication tools are available through email; web-based discussion groups are 
also conducted for each team. 

The delivery model reflects the ‘real world’ arrangements for working in the occupation of systems 
analysis, where teamwork is a key feature. Thus the delivery mode attempts to simulate a project 
team environment. Within the toolbox there is a virtual company called ‘The Marketplace’ and 
students play the role of consultants. 
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learners 



Learners are in the second stage of a four-stage program, involving the completion of certificate III 
and IV competencies. The profile of the learners is mixed with approximately two-thirds full time, 
and one-third part time. There is an approximate ratio of three males to one female student. 

Assessment design and methods 

The assessment for this program was designed at the time of the production of the toolbox and 
involved a small team of writers and content experts, using the training package competency 
standards. In the assessment design process, the team looked closely at the work of systems 
analysts with particular reference to the phases of system analysis. 

Factors influencing the choice of assessment methods include the following: 

The competencies 

The Certificate IV in Information Technology (Systems Analysis) lends itself to a problem-based 
work team approach. A range of ‘methodologies’ is used in industry for the field of system analysis 
and, to reflect industry practice, it is important to use one of these in any training or learning. The 
focus is constantly on what the students are expected to do in this role and on the performance 
level for this qualification. As the emphasis in the competencies at this entry level is on 
‘contribution’, the designers of assessment tasks have developed assessment tools that reflect this 
level of performance. 

Team-based learning and assessment 

The approach to assessment reflects the team-based structure of the learning environment. 

Students have both team and individual responsibilities and assessments. In the team discussion 
area, students submit their individual and final contributions so that everyone in that team has 
access to all the information produced by the team. Each team then submits one product (the 
development documentation and production of a prototype for solutions designed by the team) for 
the major assessment. 

The learning program and assessment plan for the teams involves six deadlines or milestones. 

These are points at which ‘deliverables’ must be submitted to the supervisor (electronically or face 
to face). Achievement of these is important because of the impact on subsequent learning and 
completion of the project. 

Feedback 

Recognition of the role played by feedback in learning is an important consideration in the 
assessment design. Continuous feedback allows learners to refine their assessment projects over 
the duration of the course and then submit a portfolio on completion. This approach to 
assessment aims to increase the learning opportunities and to encourage improvement. 

Conducting assessment 

Assessment methods 

Two of the ‘deliverables’ are produced by the individual student, and four through a team process. 
Formative assessment activities are also integrated with the learning resources that must be 
submitted; for example, a log sheet used by the learner to track and note all contributions and 
activities provides a wealth of information to the supervisor for feedback and discussion of 
progress and assessment of the team dynamics. 
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learner needs and support 

Strategies to support learners include: 

Sr an initial orientation meeting for all students 

Sr information about assessment provided on WebCT before the student commences the 
program 

^ support sessions available for any component of the program advertised on the web 
Sr communication with students through bulk emails 
Sr regular team meetings 

access to webCT from both home and campus 
Sr electronic presentations loaded on WebCT 

^ additional references or tasks posted to common areas of WebCT. 

Issues for assessment in flexible delivery 

The role of the training package 

The designers are conscious that the learning has to reflect the phases of systems analysis. They 
voice the view that the training package for this qualification identifies units of competency that 
tend to promote a ‘discrete’ skill or work area, rather than the integrated and continuous approach 
in which these competencies are demonstrated in the workplace. 

Providing models for assessment 

As students come to this entry-level course with no prerequisites, there is a need to provide models 
for assessment. The designers address this by presenting students with partly completed 
assessment tasks; their completion of the missing sections demonstrates their contribution. 

Working in teams 

The ideal learning and assessment approach for this qualification is a team-based approach. The 
provider identifies this learning model as having its own particular challenges. For example, cross- 
cultural issues can affect participation in teamwork. A variety of strategies has been used to create 
‘compatible’ teams, including students completing personality tests. 

Response rate / customer service standards 

Student expectations have increased significantly in relation to the response rate for email and the 
provision of feedback. In order to provide realistic customer service, the registered training 
organisation has set a ‘customer service’ standard to guarantee a response within a specified period. 

Work placement! experience 

The industry sector is not an easy one for the purposes of work placement/ experience for two 
reasons: 

Sr A large number of students are enrolled in the course. 

Enterprises generally seek skilled and work-ready people. 

Authenticity of evidence 

Issues around authenticity of the student’s work may arise if the program was placed completely 
online. Additional telephone or web camera communication with students may decrease the risk 
management of the authenticity issues for fully online students. Currently, team meetings are held 
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in a face-to-face mode which helps validate student performance. Quality assurance of the 
qualification is an important issue for the registered training organisation: ‘Industry has an 
expectation that when we sign off on a competency we are saying that the student has the skill’. 
One solution, bringing people to a supervised location and conducting a written exam, might 
provide some of the evidence, ‘however this would be artificial because it would not be a reflection 
of real workplace competency’. 

Challenges of the assessment approach 

Workplace settings present a challenge. Students who wish to apply the whole process to their own 
workplace would need to establish a team with colleagues. 

Strengths of the assessment approach 

The strength of the assessment approach is that it creates real and meaningful assessment activity. 
The assessment is not artificial. 

The assessment uses a collaborative approach and develops peer assessment. It reflects team-based 
workplace practice. Opportunities for both individual and team-based assessment are integrated in 
the assessment design. 

Students are provided with the opportunity to resubmit their assessment tasks so they can reflect 
on and respond to feedback and this is then reflected in the final assessment submitted. 



Case study 2: Linking business needs, learning and assessment 



Site 

Qualification 

Code 


Qantas College, Queensland 

Certificate IV in Airlines Operations (Leadership and Management) 
ZQF 40100 


Units of 
competency/ 




Provide leadership and support 


modules 




Lead and participate in work teams 






Provide feedback on performance 






Provide performance counselling 






Monitor work conditions 






Implement equal employment opportunity policies 




(Plus two elective modules.) 



Course context or background 

QANTAS College is an enterprise-based registered training organisation delivering a range of 
qualifications from both its enterprise training package ZQF 00 and other training packages. In 
response to the need to provide learning arrangements for over 30 000 staff based in Australia and 
around the world, flexibility is a key consideration in training arrangements. The Certificate IV in 
Airline Operations (Leadership and Management) is achieved through completing six core units 
and two electives and it is the first of two frontline management qualifications (QANTAS 
Supervisor Development Program). 

Delivery arrangements 

The college offers the choice of a variety of delivery modes, with a more recent focus on online 
learning arrangements. Learning arrangements include: 

te open learning centres 

workshop and classroom-based courses 

4- online learning courses. 
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Learners have a choice from totally online, self-paced learning to classroom-based learning or a 
mixed delivery mode. 

Some of the online courses are conducted through a web-based ‘virtual’ airline operations 
environment, a fictional airline company called Odyssey Airways. In this simulated environment, 
learners access learning resources, learning activities and assessment activities. Tutors are assigned 
to each group of students, and a timeframe is established for completion of the module. 
Assessment activities are based on real workplace issues that involve students in doing something 
in their workplace or writing a report in the context of their situation. 

A popular learning arrangement chosen by employees is the combination of face-to-face workshop 
activities for the ‘people management’ units and the online delivery mode for the legislative-based 
units concerning areas such as equal employment opportunity and occupational health and safety. 

Learners 

The primary audience for this registered training organisation’s VET courses is its own employees. 
Students come from a range of team leader and supervisory roles across the different business 
units and functions of the organisation (such as a team leader in a call centre, a leading hand in 
ramp baggage handling, tarmac operations, information technology services, finance operations or 
flight catering etc.). Learners may be in mixed groups with people from different units or in a 
specific group related to their business unit. 

Assessment design and methods 

The assessment for the online course was designed at the course development stage, using 
assessment guidelines from the training package. However, as these guidelines are generic and 
broad in their application, a considerable level of work by a team of content experts, instructional 
designers and learning and development staff went into the design of assessment tools to support 
the delivery of the qualifications. Assessment tools were then tested on learners in QANTAS 
programs. 

All of the assessment activities are built into the online program and course tutors can design 
further assessments if the need arises. 

Assessment approach 

Key features of the assessment approach include: 

A Some assessment activities are created in the simulated Odyssey Airlines environment. 

A Assessment tasks are embedded in the learning activities and are sequential, so as to support 
learning outcomes: ‘It’s a case of building upon the knowledge, linking it to the workplace, 
linking it to the Odyssey Airways environment — a smooth transition from the learning phase 
to the assessment phase — assessment doesn’t come as a surprise’. 

A Assessment content is based on the workplace, and the core modules focus on workplace 
needs. 

A Workplace projects are used to make the assessment practical, meaningful and relevant. 

A Formative assessment activity is conducted online using self-assessment ‘point and click’ and 
multiple-choice questions; these activities are used to assist students’ reflection on their 
learning. 

A Timeframes are set for the completion of units/modules and the associated assessments. 

A A well-developed learner management system administers, tracks, reports and monitors 
progress and achievement. 

tc Online registration for the recognition of prior learning is available. 
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Conducting assessment 

The major summative assessment methods used in this program include: 

-y- case study — written responses 

workplace projects (for example, develop a needs analysis, write a report) 

Se written assignments. 

Formative assessment uses computer-based selection-type activities, where students complete 
multiple-choice questions, matching activities and true/ false questions. 

Judgement of the evidence submitted by learners is made by the online assessor/tutor assigned to 
the particular module (each of whom has a Certificate IV in Assessment and Workplace Training). 

Learner needs and support 

The range of strategies supporting learners includes: 

-y- a mandatory face-to-face induction day to outline the program, to discuss the learning 
outcomes and assessment activities and to issue a program tool kit 

-y- documentation and face-to-face or online discussion of all assessment activities and 
information 

-y- the provision of language, literacy and numeracy support at some learning centres through the 
Workplace English Language and Literacy Program (also available for online support) 

-y- learner management: program co-ordinators are responsible for contacting, monitoring and 
assisting the learners, each of whom is assigned to a tutor. 

Issues for assessment in flexible delivery 

Explaining competency-based assessment 

The organisation is careful to promote the understanding of assessment as demonstrating what 
you can do, what you have already learnt and then how you would apply it’. This is in response to 
the perception of some learners, who had not been through a ‘formal’ assessment process for a 
long time, of assessment as ‘tests’ or ‘exams’ and also, the experience of failing. 

Time management 

QANTAS College recognises the issue of supporting distance learners in assessment and has 
responded to feedback from learners, particularly in relation to time management and information 
about ongoing course requirements. The registered training organisation has learnt from its 
experiences in online delivery and assessment and has now put in place strategies to address these 
issues. 

Through early evaluations they identified a need to prepare learners from the outset with 
comprehensive information about course expectations and assessments. Part of this review has 
involved the design and development of more detailed study guides. 

Communication and learner expectations 

The organisation has also discovered that the impact of the tutor in communicating with students 
is a critical issue in online learning: ‘Using email and quickly providing feedback on learning and 
assessment are important issues in distance environments’. There is a key role for a registered 
training organisation to manage learner expectations in an email-based online learning 
environment. With the speed of communication and the ‘anytime and anywhere’ capacity of email 
submissions, the organisation acknowledges that students expect a very fast turnaround time. 
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As part of the professional development commitment to the program and to support its tutors and 
assessors, the organisation holds two major meetings a year with them. 

Strengths of the assessment approach 

4- Assessment is promoted strongly as a way of demonstrating what you can do and what you 
have learnt in your work and course. 

4- Assessment is linked to real workplace jobs and tasks. 

4- The tutors and assessors complete self-assessment audit checklists to quality assure their 
assessment processes and are issued with copies of the assessment principles. 

4- Regular analysis of learner course evaluations helps review assessment activity. 

4- The learner management system supports and monitors student progress. 



Case study 3: Immediate feedback online 



Site 

Qualification 


Box Hill Institute of TAFE Victoria 

Certificate III in Education (Office Support) Traineeship 


Units of 


NOS 124 mapped to FIN 301-05 


competency/ 




modules 





Course context or background 

Box Hill Institute of TAFE in Victoria offers the Certificate III in Education (Office Support), 
covering 1 1 compulsory modules and six office electives. Competency may be achieved through 
recognition of prior learning. 

The module NOS 124, Accounting to trial balance, is one of the office electives for the program. 
The online learning and assessment package for this module was developed in response to a 
request from the off-campus centre of the institute to provide flexible delivery arrangements. 

Delivery arrangements 

The complete qualification is offered through a mix of distance learner guide resources and online 
delivery. The online module NOS 124 has been closely mapped to the relevant modules of the 
BSA 97 training package competencies. The course is designed as eight units, three of which are 
almost ‘stand alone’ (for example, Payroll, Cash) and five of which are integrated and 
interdependent. 

The accounting modules of the Certificate III in Education (Office Support) are offered online 
(Accounting online). Students enrol and have 18 weeks of support scheduled from the time of 
enrolment. Currently, students can enrol in the program at any time (although these arrangements 
may be revised at a later date, when continuous enrolment will be replaced with monthly student 
intakes). 

The program is ‘self-paced’ and learners can access a range of online resources as well as a 
textbook used widely by Victorian TAFE. 

Learners 

Learners are generally educational support staff performing clerical and administrative roles in 
primary or secondary schools. 
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Assessment design and methods 

Assessment was designed at the time the program was developed by an experienced teacher of 20 
years and external assessor in the accounting field. 

It was decided from the outset that while a number of the assessment tasks can be completed 
online and submitted to the tutor, two of the tasks core to the achievement of the module are to be 
conducted in a supervised environment. The reason for this approach relates largely to authenticity, 
ensuring that the student completing the assessments is the enrolled student. Their design is such 
that students need to have completed the earlier online tasks to successfully complete the two 
supervised assessments. 

Expert users (distance learning and business administration tutors) were involved in the 
development of the packages and student feedback was built into the development stage. 

Assessment approach 

The assessment approach is one of continuous assessment to support students in their learning. A 
significant consideration in the design of the assessment is that there is no final exam. 

Other features of the approach to assessment include: 

-y- Students identify when they are ready for assessment so flexibility is enhanced through a 
system that does not prescribe specific dates for the actual assessment events. However, all 
assessments have to be completed within the prescribed support period. 

^ Students who do not comply with the requirements receive a ‘not competent’ result and have 
to re-enrol. 

The registered training organisation has an integrated student learning and assessment tracking 
system. 

^ Practice assessments are a feature of the program and completion of these events, forwarded 
to the tutor, is mandatory before formal assessment can be undertaken. 

Formal assessments are completed online and forwarded to the tutor who assigns a score. 

-y- Practice and formative assessment are integrated with learning activities. 

-y- Technology is used to enhance immediacy for assessment; for example, email is used to 
forward assessment. 

-y- Supervised assessment is a feature for two of the assessments (signed off by a supervisor in the 
workplace if the assessment is not conducted at the campus). 

Conducting assessment 

There are eight assessment events, of which six are online and two supervised. 

At the end of each tutorial the students complete short self-checking quizzes comprising multiple- 
choice and true/ false questions, completed and marked online. Immediate feedback is provided 
and students have the opportunity to resubmit until they get it right. 

Skill assessment is also conducted online. These competencies are very practical; for example, the 
demonstration of inputting data into software and analysing data to produce spreadsheets. 

Flexible assessment of trainees in the program draws on the specific workplace context for each 
trainee. 

Preparation for assessment tasks 

There is a well-developed process of preparation for assessment. Students download the task, 
complete it offline, log on to the course, use the solution data from the exercise and email solutions 
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to the tutor. The tutor provides feedback and, if satisfied, the student then attempts the formal 
assessment. 

The two supervised off-campus assessment tasks are sent manually to the nominated assessment 
centre (other campus, university, school or workplace etc.) where supervision processes are in 
place. This assessment serves to ‘validate’ the other assessments to a certain degree, in that students 
cannot complete this assessment without having completed the previous ones. (At least one 
student at a remote location has commented, understandably, that they would prefer the process to 
be online.) 

Evidence 

Evidence is submitted to the course tutor both electronically and by mail. The course tutor 
determines if the student achieves the competencies. 

Learner needs and support 

Assessment information is provided to learners on enrolment through an initial email contact by 
the tutor or assessor. Here tutors introduce themselves, give details for a study plan and explain the 
18 weeks of support arrangements. Information about assessment is also set out in the learning 
guides. Students enrolled under traineeships receive substantial monitoring and the co-ordinator 
plays an active support role, intervening as needed. 

To support assessors, a review process is in place to inform changes and improvements to future 
delivery. 

Issues for assessment in flexible delivery 

Completion rates 

The completion rates of flexible delivery students in this course are being reviewed. There is 
concern over the attrition rate in flexible delivery arrangements, both in the distance and online 
environments. 

Studying at a distance is hard — too hard — not the content of the course, not the organisation 
of the course — just studying in isolation. 

Timeframes for submission of assessments 

Without timeframes for submission of assessments, students tend to leave them to the last minute. 
This creates pressure on the teacher and contributes to the attrition rate. In the review of this 
online program there is a recommendation to introduce more structured timeframes for 
completion and submission of assessments. 

Responding quickly to time management issues for students is a challenge — learning in this 
mode you need to be able to manage yourself. There is a need for more intervention. 

However the resources do not always make this possible. 

Peer support 

Peer support is a difficulty in distance and online modes of learning. Online technologies need to 
be exploited to their fullest to attempt to overcome student isolation, although as one tutor 
commented: ‘regardless of the technologies, it’s the same issue — that of isolation for distance 
learners’. 

Strengths of the assessment approach 

The immediacy of feedback and the use of technology supports the use of assessment online. 
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The online potential for far greater student interaction through forums makes this mode very 
attractive for future distance education because of the possibilities of student communication. 

The facility exists for students to study at their own pace and identify when they are ready for 
assessment. 



Case study 4: Sharing responsibilities; making assessment accessible 

Site West Coast College of TAFE 



Qualification 

Units of 

competency/ 

modules 



Certificate II in Food Processing: FDF 98 
FDF COROHS 2A 



Course context or background 

The West Coast College of TAFE in Western Australia delivers a food processing qualification on 
site to the employees of a major food processing enterprise through its Business Development 
Unit. The program began with a Workplace English Language and Literacy Program grant to 
support and develop workplace communication skills in preparation for training and skill 
recognition under the national Food Processing Training Package. 

Delivery arrangements 

The delivery arrangements for the enterprise-based training are mixed mode, with on-the-job, one- 
to-one and small group components. 

Assessment design and methods 

The training package assessment principles inform the planning; that is: ‘The competencies 
described in the unit need to be performed over time and events, under normal workplace 
conditions, having due regard to the key assessment principles of validity, reliability, fairness and 
flexibility’. 

The training delivery approach was developed as a co-operative effort between the registered 
training organisation consultant, key managers and supervisors. The assessment system was 
designed to support the training system. With the assistance of a consultant from the National 
Food Industry Training Committee, the assessment system was reviewed and changed in the 
middle of the year. By this stage a number of assessors were trained and could provide input into 
how the assessment process would ultimately be delivered to the workforce. 

The involvement of the enterprise in the development of the training and assessment system was 
imperative. In this way it meets the needs of the diverse group of people who use it. Using the 
model suggested by the National Food Industry Training Committee, assessment tools were 
developed by a team of people including the occupational health and safety co-ordinator, the 
quality manager, the area assessors (and unit ‘experts’) and staff members from the registered 
training organisation. 

The workplace assessors tested the new assessment tools and the changes they suggested were 
incorporated in a revised format. 

The assessment approach has to suit, it has got to fit in with the enterprise. 

(Registered training organisation consultant) 

With the needs of the enterprise in mind, the assessment approach adopted is on the job 
and one on one. 

This approach is the least intrusive and the most informative. 

(Registered training organisation consultant) 
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Factors influencing the choice of assessment methods include: 

Assessment planning and design must take in a range of workplace issues, including 
production schedules and shift patterns. 

<■ Learning and assessment resources, largely developed by the organisation, need to be worked 
into the context of the enterprise’s processes and procedures. 

-y- When considering the appropriateness of using flexible delivery modes in the enterprise, the 
use of technology was rejected as online training and assessment discriminates against workers 
who do not have access to computers or who are not computer literate. It also raises issues of 
fairness with workers from a non-English speaking background. 

Conducting assessment 

Assessment methods include: 

<■ direct observation of performance by assessors 
-y- questioning of candidates by assessors 
role play 

<■ written reports 
^ case studies. 

Underpinning knowledge is predominantly assessed through verbal questioning. Dimensions (such 
as task contingency, task management etc.) are built into the assessment tool. 

Learner needs and support 

Worker support 

Strategies in place to support learners include: 

Information provided informally by workplace assessors; briefing sessions conducted with 
candidates prior to assessment events; and formal notification to the candidate by letter 
regarding assessment information. 

Negotiation of assessment times to ensure reliability and fairness, given that learners work 
long, physically demanding, shifts; conducting an assessment at the end of the shift, for 
instance, would not be fair nor result in ‘representative’ performance. 

Support for learners with non-English speaking background language needs and for those with 
literacy needs via Workplace English Language and Literacy initiatives. 

The support of the workforce for the assessment process grew during the year of Workplace 
English Language and Literacy funding when most of the workers took part in off-the-job 
communication skills training. 

Through regular interactions with the registered training organisation, trainers using adult learning 
principles in the delivery of the training, trust has grown and with it a culture of learning in the 
workplace. Assessment is seen as a natural extension of training and any concerns about sitting 
exams or failing have been allayed. 

Issues for assessment in flexible delivery 

The assessment process has undergone several revisions and is in a format that suits both assessors 
and workers. New assessment tools are being developed as work instructions for each area are 
reviewed and developed. 
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The assessment method is holistic with as many units of competency being addressed as are 
relevant to the job. In all the assessments, the core modules of safety, food safety, communication 
and quality are incorporated. Separate assessments for the core units have also been developed but 
only address those aspects of the evidence guide not covered on the job. The holistic approach has 
significantly reduced the number of assessments in which the worker has to participate. 

Flexibility in assessment is achieved by taking into account the timing of assessment (to fit in with 
worker’s shift patterns and production requirements) and the language skills of the participants (by 
providing an interpreter where appropriate, or by changing the assessment methods from questions 
to demonstrations where practicable.) 

Strengths of the assessment approach 

Inclusiveness is a major strength of this assessment system. From the beginning, there has been a 
strong emphasis on ensuring that the process is accessible to, and considerate of, all learners. In the 
registered training organisation consultant’s words: 

Flexible delivery and assessment means that you do what is necessary in training or 
assessment — in order that everybody and anybody is able to access it. You still want everyone 
to have the same opportunity whether they’re on site, off site or 10 000 km away. So it means 
looking at the way you do things to make sure that you are inclusive. 

There is a strong understanding that the learners are undertaking a national, competency-based 
qualification. The concept of competency is well understood and often discussed, particularly in the 
planning of the whole-of-job assessment approaches (as opposed to the assessment of individual 
units of competency). 

The principle of reliability is supported through professional development of assessors, regular 
meetings with staff trainers and assessors and quality checks of assessment evidence. 

The assessment system for the workplace has been documented to ensure consistency. 

Case study 5: Meeting local industry needs 

Site Hunter Institute of TAFE NSW 

Qualification Certificate III in Boat and Ship Building 11756 



Course context or background 

The Hunter Institute of TAFE delivers one of only two courses for boat-building apprentices in 
New South Wales. The program is conducted at a customised open learning skill centre, where 
facilities support both practical skill development and the acquisition and application of 
underpinning knowledge. Considerable use is made of resource-based learning and a learner- 
centred delivery mode. 

The delivery model was developed five years ago in response to the need to take account of 
industry requirements and demands. Extensive consultation with the industry sector was 
conducted to develop an industry-responsive mode of delivery and assessment. 

As one initiative towards flexible delivery, the learning centre conducted a pilot program using 
online and video-conferencing technologies to deliver its program to apprentices based in seven 
enterprises in Taree. The course module used for the pilot online program was Hull design 
calculations. 

Features of the flexible delivery include: 

-y- flexible attendance patterns to fit in with workplace needs and ‘production’ demands of the 
apprentices and their employers 
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Sr choice of learning modules 

Sr choice over the sequence of learning modules within the course. 

Delivery arrangements 

The program is delivered over three years. There are 30 modules in the course and on completion 
of the mandatory module, introduction to boat building, learners can be involved in any one of these in 
a sequence of their choice. 

Learners 

The learners are either apprentices, seekers of apprenticeships or unqualified shipwrights. 

Assessment design and methods 

Curriculum developers and teachers at the learning centre designed the assessment for this 
program when the flexible delivery course was first developed. Considerations at the design stage 
included the timing of assessment and the weighting given to assessment tasks. Syllabus documents 
and state curriculum guidelines were used in the development of the assessment. Amendments 
were made to the design of the previous assessment module to reflect the design of the flexible 
course material. 

The approach to assessment includes considerable input from industry. All assessment is currently 
conducted at the learning centre, and students negotiate the timing and their readiness for an 
assessment. 

Conducting assessment 

Assessment is conducted through: 

^ observation 
written tests 
-y- drawing tests 
Sr practical workshops 
Sr project work for integrated assessment. 

The evidence collected relates directly to the learning outcomes for the course. The teacher and 
assessors use marking grids and checklists to record evidence. 

Learner needs and support 

Support for learners includes information about assessment at enrolment and during the 
compulsory prerequisite module, introduction to boat building. The program directs learners through 
the self-paced learning system and sets out the information about assessment, which is included in 
all the learning. 

Issues for assessment in flexible delivery 

The time available for practitioners is a constraint. There is a need for dedicated time for 
assessment development, particularly when a review of the training and assessment methods is 
being conducted. 

The pilot program for online delivery raises a number of learning design issues: in particular, that 
of making very explicit all the requirements and information the student needs. In one tutor’s 
words: 
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There are so many unknowns for the student sitting in front of a computer screen by 
themselves. You have got to really lay it all out for them. 

There is a diversity of trade areas or specialist industry sectors across the employers. In future, 
considerations for online delivery, on-site workplace assessment will have to be built into the 
assessment approach and systems. 

The boat-building learning centre is set up so apprentices can demonstrate their technical and 
practical skills. The extent to which workplace performance evidence can be used is yet to be 
explored; however, learning centre staff recognise that this would need quality assurance processes 
and involve additional resources and time. 

There is a particular interest in the possible use of computerised assessment item banks for the 
theory components of the course. It is anticipated that this would increase the reliability of 
assessment, and ultimately provide better support for learners by freeing up marking time. The 
developers agree, however, that any computerised assessment system must allow assessors to see 
how a candidate arrives at a solution. 

Strengths of the assessment approach 

The strengths of the assessment approach are the flexible and responsive system for the learner 
and their employer, as well as learner involvement in planning timing for assessment, which places 
more responsibility on the learner (within a supportive environment). 



Case study 6: Weaving in assessment principles 

Site Torrens Valley Institute of TAFE SA 

Qualification Diploma in Engineering (Computer Systems) 

Units of NE029 BKMB Electronic hand soldering technology 

competency/ K ey competencies 



Course context or background 

The Diploma in Engineering (Computer Systems) is delivered by the Faculty of Electronics and 
Information Technology, Torrens Valley Institute of TAFE, South Australia. The campus has been 
designed to deliver the qualification through a variety of modes, from traditionally structured face- 
to-face teaching to fully self-paced independent and flexible learning arrangements. 

The flexible delivery mode used at the institute is supported by the centre’s facilities, a long-term 
commitment to a learner-centred perspective in education, and a faculty with a team-based 
approach to course delivery. Assessment of the key competencies has been integrated into the 
practical and theoretical skill-based learning in the faculty through a formal recognition process. 

Delivery arrangements 

The key features of the flexible delivery mode include: 

resource-based learning with a strong focus on using technologies (computer based, video and 
print) 

progressively introduced and supported self-paced independent learning 
-y- mentor/ coach appointed for each learner 
4- tutorial support provided by the team members 
te integration of key competencies in learning arrangements 



64 



Exploring assessment in flexible delivery of VET programs 




4- remote learner link-up 

4- learner access both on campus and at home. 

Learners 

Learners include those who want to upgrade skills for career or work purposes, to re-train or to 
satisfy a particular interest. At entry, all learners complete a quiz that is used to determine their 
mathematics and reading skills and the level of support they need to participate in a resource-based 
learning program. 

Assessment design and methods 

When the college introduced the flexible learning mode, an important consideration in the 
assessment design was the need to reflect as much as possible the workplace demands of the 
industry sector. Assessment was designed using a team-based consultative approach with reference 
to the principles of assessment, the national module descriptors and the local program guidelines 
(derived from the state guidelines). 

The design of the assessment was guided by the specific performance criteria for all the learning 
outcomes. Integration of the assessment design with the flexible learning arrangements was an 
important consideration. 

Assessment of key competencies 

The registered training organisation has designed and developed an assessment process for formal 
recognition of key competency areas. This is reflected in the way learners manage their learning, 
including their timetables, and communicate with staff and other learners in the centre. The flexible 
learning environment creates the opportunity for key competencies to be developed and assessed. 

Technology-supported assessment 

Each module of the course is assigned a module facilitator who is responsible for maintaining the 
assessment and updating content and assessment methods and tools. 

The design also considers the processes for graded assessment. During development, it was 
decided to progressively maximise the use of technology, so that now a computer-based learner 
and assessment management system which allows tracking of learner progress is in place. 

Technology-supported, remote assessment is a feature of a pilot program for apprentices 400 
kilometres from the registered training organisation. Using an internet connection, soldering 
boards and web-cam equipment, learners are able to demonstrate the practical skills they have 
developed as simulation of the workplace skill. 

The organisation recognises the importance of well-specified performance criteria allowing for 
flexibility in assessment, while ensuring the validity and reliability of the assessment. This includes 
negotiation with the learner about the best method to meet their needs. For example, if a learner 
has difficulty using computer-based assessment activities, there are print based or verbal 
questioning alternatives. 

Some of the more traditional assessment methods, such as short-answer written responses and 
multiple-choice questions, are used to assess underpinning knowledge. These are provided by 
computerised assessment — a test bank — that can generate randomised, multiple-choice questions, 
short-answer questions and graphical items. 

One of the considerations for the development of computerised test item banks is cost- and time- 
effectiveness. The registered training organisation considers the initial development and subsequent 
maintenance costs as an investment, freeing up lecturer time to support the learning process. 
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Promotion of key competencies assessment 

Participation in the recognition of key competencies is a voluntary process. The three performance 
levels specified for achievement of key competencies are: 

-y- doing: following a given model 

Sc managing: demonstrating choice of strategies 

Sc creating or shaping: doing something of your own or taking an existing process or product and 
shaping it into something else. 

The learner chooses the performance level, and determines how evidence will be presented to the 
assessor. The module facilitator then assesses the candidate. Options are provided for candidates to 
present their evidence in a range of ways. For example, a learner may provide evidence by videoing 
their performance. Because recognition of achievement of the key competencies is a voluntary 
process, this competency assessment has to be promoted. 

Features of the assessment approach 

In summary, features of the assessment approach include: 

Sc presentation of explicit and ‘up-front’ information about assessment in the learning guides 
simulation of real workplace-based activity 

Sc use of information communication technologies for computer-generated knowledge tests and 
assessment 

Sc timing of assessment chosen by learners 
Sc negotiation of assessment methods by the learner 

-y- integrated assessment of key competencies with practical skills and theoretical knowledge 

-y- tracking learner progress through a computer-based learner management system 

Sc supporting learners by providing opportunities to attempt assessment activities more than 
once. 

Conducting assessment 

To collect evidence, assessment is conducted through: 

-y- computer-mediated ‘supply-type’, short-answer, multiple-choice and true/ false responses 
Sc practical experiments and skill demonstrations 
Sc written reports 

Sc demonstration of practical activity or operation of equipment 
T research reports 
^ oral presentations 

Sc self-checking computer-based questions 
-y- workplace project (if requested) 

Sc feedback during practical activity 

Sc monitoring through computer-managed learning system. 
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For the soldering module, assessments include: 

-y- assessor observation of the learners’ soldering skills performance in the workshop, using an 
assessment checklist of industry standards 

learner inspection of a sample of a soldered assembly and verbally identifying the good and bad 
features of the assembly, providing reasons and explanations. 

Computer-assisted assessments can be completed at a time chosen by the learner. Candidates book 
in for a test, log on to the computer at the appointed time and complete the assessment in their 
own time. A range of strategies is in place to support those who are unsuccessful in their attempts 
at the tests. 

learner needs and support 

Orientation 

Learners receive information about the assessment process in an orientation that overviews the 
whole of the flexible learning arrangements and explains assessment. A system of ‘scaffolding’ is 
then provided in the initial stages, mainly through the assignment of a mentor or key contact 
person for every learner. The mentor monitors progress, adds structure where required, negotiates 
meetings and offers general support and advice. 

Formative assessment feedback 

A variety of formative assessment tasks are incorporated in the learning activities. These self-check 
exercises present opportunities to support the learning process. Where learners make mistakes, the 
responses take the learner back to an explanation of the correct response. The approach to 
feedback is underpinned by the philosophy that learners are most motivated for learning when 
being assessed. 

Other forms of learner support 

Other forms of learner support include tutorial and individualised support for subsequent attempts 
at assessment tasks, and scheduled workshops. 

Support for particular learning groups is provided at the centre. For example, using a signer for any 
audio material training and assessment for hearing-impaired learners is provided. Technology is 
used for distance or remote assessment of apprentices in Port Lincoln. 

Learners are also supported through chat rooms and extensive use of email. 

Issues of security a nd o nline assessmen t 

Learners are assessed on site, in a secure room. Learner ID cards are barcoded and swiped at the 
time of the assessment event. This is part of the procedure for the integrated learning and 
assessment management system. Currently, online delivery and assessment is not really an issue for 
the registered training organisation as the flexible delivery mode is a self-paced, campus-based 
arrangement. However, web-based assessment has been piloted with off-site learners. 

Issues for assessment in flexible delivery 

Challenges include: 

-y- developing strategies for dealing with system-level requirements: initially the registered training 
organisation had to work out how flexible delivery arrangements complied with statewide 
recording systems that require start dates and finish dates 
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managing learning in a self-paced learning environment: staff are required to respond to a 
range of learner needs, at different times. Making sure assessment is completed is part of this 
learner management process, of which the use of a computerised system is a key feature, while 
some learners need to be supported with structure and frameworks 

finding a way to recognise and demonstrate over and above the basic levels of the key 
competencies: the registered training organisation is keen to investigate this issue as is industry, 
because employers value the key competency skill areas as much as they do the practical and 
technical skill areas. 

Strengths of the assessment approach 

Strengths of the assessment approach include: 

^ workplace relevance of the assessment tasks (assessment principles are ‘all woven into what we 
do’) 

-y- self-checking assessment 

-y- a well-developed support system for assessment 

provision for evidence collected in the workplace, through quality-assured processes 
a process for moderation of assessment across the team members 
constant checking with industry standards ensuring workplace relevance 
regularly conducted professional development activities. 

Case study 7: Effective and responsive assessment 

Site Australian Drilling Industry Training Committee 

Qualification Certificate III in Drilling (Mineral Exploration) DRT 30498 



Course context or background 

The Australian Drilling Industry Training Committee conducts training and assessment services 
primarily for small-to-medium-sized drilling enterprises across Australia and is responsible for the 
development of the competency standards. Its major activities include the recognition of the skills 
of existing workers in small- and medium-sized drilling operations. 

People working in this field are mobile and field based, and the nature of the work means they are 
frequently located in remote areas. Where training is required, it is delivered through a distance and 
mixed mode, involving on-the-job and workshop format (for larger organisations). 

While some training and assessment has been conducted for those wanting to enter the industry 
sector, the majority of activity at present is for those currently employed in this industry sector. 

Assessment design and methods 

The assessment approach has been guided by the training package guidelines. These suggest the 
type of evidence required and a range of methods, including checklists for performance and open 
questions. The support resources from the training package include assessment tools and materials 
for the assessor and candidate. 

The features of assessment have been influenced by a number of factors, foremost of which are 
the nature of the industry and the needs of the enterprises. The skill areas for this qualification are 
in a strong ‘hands-on’ set of competencies. 

As the assessment is workplace based, on-the-job training and assessment are necessary; a qualified 
workplace assessor conducts the assessment. Assessment is negotiated with the employer for 
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auspicing arrangements. There is no grading in the assessment system, the candidate is determined 
as either ‘competent’ or ‘not yet competent’. 

Conducting assessment 

A major consideration in the choice of assessment method is the experience of the individual being 
assessed. 

The assessor conducts an initial conversation with candidates to collect background information 
and determine an appropriate strategy (either on-the-job performance assessment, recognised prior 
learning, or a combination of both). 

Holistic assessment involves a site visit to the drilling rig, to observe the driller’s performance over 
a period of time, using a checklist for assessment. Underpinning knowledge is assessed with on- 
the-job questioning, using ‘what if situations and scenarios to assess troubleshooting 
competency — addressing task management and contingency skills. A third party, such as a drill 
supervisor, may sometimes gather evidence. The evidence is then considered and a judgement 
made by a qualified workplace assessor and, where necessary, third-party statements are collected 
from the client’s supervisors or representatives. These are documented and signed. 

Written assessments of underpinning knowledge are used for distance learning. For example, for 
the distance module on Rig mechanics, the learner may be required to sketch an item of equipment 
for which they are responsible, labelling all parts and then developing and documenting a 
maintenance schedule for the item of equipment. 

Legislative demands 

Legislative requirements for mining in Queensland are a driving force behind the implementation 
of competency-based training and assessment. Contractors in the mineral exploration drilling 
sector are now often required as part of the tendering process to provide evidence of the 
competency of their personnel. 

Issues for assessment in flexible delivery 

It should be noted that there is a possible concern about consistency in assessment. With a number 
of assessors in the field, the standardised tools play an important role in maintaining confidence in 
the approach. 

There is also an issue in regard to the variance in the reporting level for evidence. Some assessors 
provide more details and comments than others, so quality can be an issue. 

Strengths of the assessment approach 

The industry is diverse and requires a flexible approach to delivery and assessment. The strengths 
of the adopted approach include the following: 

The on-the-job experience is valued. 

^ The competency standards provide an appropriate way to recognise workplace skills. 

Working with enterprises gives the assessment a context. 

-y- Standardised assessment tools are in place to support reliability, and candidates indicate their 
readiness for assessment. 
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Case study 8: Collaboration in assessment 

Site Pelion Consulting Pty Ltd 

Qualification Diploma in Frontline Management - 7042 

(Diploma of Business Frontline Management BSB 51001) 



Course context or background 

Pelion Consulting Pty Ltd works with a range of enterprises across Tasmania delivering the 
Frontline Management Initiative certificates and diploma. The qualification was developed in 
response to the Karpin report (Quality of Education Review Committee 1985) which claimed there 
was a lack of skills, qualifications and training support for frontline managers Australia wide. 

The qualification is delivered through a recognition process. The registered training organisation 
works with a range of enterprises to implement formal recognition of the competencies outlined in 
the Frontline Management Initiative. This qualification is delivered at three Australian 
Qualifications Framework levels from certificate III, up to diploma levels involving the 
achievement of six, eight, and eleven units of competency, respectively. 

The ‘enhanced’ Frontline Management Initiative package has been released. At the time of the 
interview, the registered training organisation was working with the original Frontline Management 
Initiative competencies and is now registered against the BSB training package qualification. The 
organisation is currently working with the existing competency standards and their requirements. 

Delivery arrangements 

The initial stage of the process involves bringing candidates together and working through the 
recognition process, discussing and identifying the necessary types of evidence to fulfil the 
requirements. Where gaps are identified through this process, the consultants work individually 
with candidates to set up a workplace project that enables the candidate to develop skills and 
provide evidence for the achievement of the competency. Candidates then provide this evidence to 
the assessors. An initial desk assessment of the evidence is undertaken, followed up with 
interviews, and a structured process using email and other communication tools to collect further 
evidence and verify witness information. 

An initial desk assessment of the evidence is undertaken, followed by interviews and a structured 
process using email and other communication tools to collect evidence and verify witness 
information. 

Candidates undertake the program in timeframes ranging from four months to two-and-a-half 
years. 

Assessment design and methods 

The approach used by the registered training organisation is informed by the competency standards 
and a number of resources, including the Portfolio Pro guide and handbook and the Pelion Frontline 
Management Initiative Assessment Management Kit. 

Given the nature and the context of this qualification, the assessment planning begins with the 
enterprise, involving managers and sponsors of candidates. An initial briefing session ensures an 
understanding of the competency standards, commitment to the processes, and availability of 
necessary resources. 

The assessment approach takes into account the multi-faceted nature of the qualification. It is 
generally delivered in the context of internal ‘workplace change’ agendas and therefore needs the 
involvement of managers, the sponsors and other key workplace personnel. Having an internal 
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co-ordinator in the workplace can be a key factor to the success of the Frontline Management 
Initiative as a workplace improvement tool. 

Once arrangements have been made with the workplace, the process usually begins as described 
above. 

The consultant and the candidate, together, use a documented system to build up a profile of the 
candidate’s awareness of the assessment procedure. This includes discussion of the processes and 
agreement on the terms and the systems, supported by permission forms, entry surveys etc. 

Other features of the approach include: 

planning sessions with individuals to assist in the development of their portfolio of evidence to 
be assessed 

-v> involvement of third parties as necessary; for example, in relation to the leadership 
competency, third-party evidence may be required from workplace colleagues 

^ assessment related to the individual’s particular environment 

links made, where appropriate, to performance appraisal systems 

<■ quality assurance for the collection of evidence, including statutory declarations 

-y- workplace projects used to support evidence. 

Conducting assessment 

A number of factors influence the choice of assessment methods. The registered training 
organisation establishes the enterprise’s needs, first and foremost. Another key factor is the 
standards themselves. The assessment methods are then determined according to organisational 
issues and the individual’s needs. 

Judgements are made based on a range of evidence provided by the candidates. This is 
individualised to reflect the learner’s styles and preferences. For example, a candidate may have a 
preference for mapping evidence using charting and diagrams in preference to a written document. 

Evidence for assessment is collected through: 

-y- portfolios 

questioning and interviews 
workplace projects and products 
-v> third-party evidence and testimonials. 

Learner needs and support 

Support for learners is provided direcdy to the individual, and strategically through the candidate’s 
employer organisation. 

This strategic approach requires the organisation to implement: 

<■ a Frontline Management Initiative co-ordinator position: a contact person for large groups, or 
where the client is working in isolation 

-v> an initial briefing with senior management and human resource personnel 

<■ clarification of standards for managers so they can understand the processes 

an analysis of the roles and communication channels within the organisation to support 
individual candidates to achieve competencies; for example, candidates may need additional 
workplace projects, access to certain information, or a change of roles to challenge existing 
practices or introduce new approaches. 
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Support for individual learners is provided through: 
tc information about assessment and recognition 

tc creating a more relaxed environment through audiotape interviews and conversations 
matching the styles and approaches of the candidate to his or her assessor or coach 
customising the level of support needed by the individual to the level of the qualification. 

Issues for assessment in flexible delivery 

The recognition processes 

There are various perceptions about recognition processes. For example, some employees who 
undertake traditional campus-based courses in their own time may feel it is unfair for a colleague 
involved in an initiative such as the Frontline Management Initiative to gain a qualification in four 
months. There is a perception about what is involved in gaining a qualification and it is often linked 
to the traditional time-based process, as opposed to providing evidence that is workplace related. 

The recognition process can also be confronting for those who may ask: ‘Why do I have to provide 
evidence?’. They may take offense at having to prove themselves. There is a need to work 
sensitively and slowly with these candidates, and to demonstrate clearly the benefits of collecting 
the evidence. In the words of a registered training organisation consultant: 

Once they understand where they are coming from and they realise it is actually valuable to 
have a qualification based on what you actually do at work and not what is talked about in a 
classroom, they really like it. It proves to them they are really doing a good job and it 
recognises that they are. 

Using technology 

In the earlier days of the delivery of the qualification, the registered training organisation used a 
website for chat room discussion and as a peer support mechanism. While a few of the participants 
liked it, the fact that they were working in clusters made it redundant: 

In some ways after the initial group meetings it becomes pretty singular — participants talked 
about ‘my job’, ‘the way we do it here’ and ‘what I want to do’. The need for that early group 
was not to talk online. They were in clusters so they had other people in their workplace or 
area to communicate with. This might be different if the context was different and we were 
dealing with individuals and those in isolated circumstances. 

(Registered training organisation consultant) 



Strengths of the assessment approach 

The approach to assessment has a number of strengths: 

tc The developmental work that has informed the production of the assessment resources 
included working with other providers and agencies to achieve consistency, through 
moderation groups, in interpretation of the competency standards, development of resources, 
and mapping against the standards. 

The use of assessment methods is flexible while validating work performed by the candidate. 

The approach is highly collaborative — the candidate and assessor work closely together to 
identify evidence and plan assessment, but with the individual still in control. 
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Case study 9: Solutions for a remote and small-scale registered training 

organisation 

Site West Pilbara College of TAFE WA — Karratha Campus 

Qualification Certificate III in Childrens Services (Child Care) 

Units of Provide physical care CHCCN2A 

competency 



Course context or background 

Karratha Campus is the main campus of a TAFE college delivering a range of qualifications in a 
remote area of Western Australia. The Certificate III in Children’s Services is delivered through an 
open learning centre and a distance learning program to a diverse and geographically wide region of 
Western Australia. 

The learning and assessment material used for the program has been developed through a 
collaborative arrangement between Central Metropolitan College of TAFE WA and TAFE SA. 

All participants are enrolled in the open learning program, and use purchased self-paced learning 
guides and campus learning resources for their course. They choose where they study — at home, 
work or at the campus. There is also flexible enrolment, allowing learners to enrol at any time. 

Learners are advised that 1 8 months is a typical time in which to complete the certificate III level 
and they are allowed six months to complete a unit. On completion of the certificate III learners 
can then move onto the diploma-level program. 

While learning resources are print based, learners have the option to email assignments to the tutor. 
There is a growing interest in using this media once people are confident with the technology. 

The learners 

The majority of the learners come from Karratha and surrounding towns and many work full time 
in long-day childcare centres. A number of learners have moved and are now completing the 
course at some distance from the college. Ninety-five per cent of learners are of mature age. There 
are very few school leavers. 

Assessment design and methods 

All the assessment material needed for this program is purchased. 

The team who developed the learning resources for the competency standards developed the 
learning and assessment material in 2000, in consultation with a variety of stakeholders including 
TAFE personnel, industry bodies and government departments. 

The competency standards specify that assessment must be conducted on licensed premises; that 
is, in a long-day care centre or an out-of-school care centre for six to 12-year-olds. 

Features of the assessment approach include: 

-y- strong links with industry 
-y- field-based approach 

<■ assessment of individual units of competence 
recognition of current competence 
-y- the negotiation of timing of assessment by learners. 
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Conducting assessment 

Assessment methods used to collect evidence include: 

-y- written assignments 

demonstration of skills and tasks on the job 

tr using a simulated environment (the community services classroom, for example, is set up for 
nappy-changing, bottle-making, lifting children out of cots etc.) 

verbal questioning on site during teacher/ assessor workplace visits 

observation 

third-party evidence (using an assessment checklist developed in the package, with evidence 
quality assured). 

Learner needs and support 

Information about assessment is given to the learner in a session with the lecturer prior to 
enrolment, and in a self-explanatory learning guide. Referrals to reading and writing courses for 
literacy skills development are made and lecturers organise work placements. Learners notify the 
lecturer when they are ready for assessment, and the lecturer assesses them on site. 

Issues for assessment in flexible delivery 

The course lecturer has identified a number of issues and challenges that relate to assessment in the 
flexible delivery of the Children’s Services Certificate. These are described below. 

Selection and use of a range of assessmen t methods 

In the context of the selection and use of a range of assessment methods in a distance mode, in a 
face-to-face class, for example, it is often easier to use a verbal questioning technique. It is difficult 
to support the group learning process in a distance learning mode. Comments include: 

I can easily set up a situation in which learners learn in groups and then I can assess their 
learning directly. 

Larger colleges with more enrolments can cluster units of competency and then assess a 
number of competencies through an assessment activity. In a smaller college it is more 
practical to assess units of competence on an individual basis. 

Time and resources required to develop original materials 

Small remote registered training organisations do not necessarily have the time and resources to 
develop original learning and assessment materials. Purchasing a quality packaged resource that 
includes the assessment tools and can be customised is one solution. At Ivarratha this has been a 
practical solution; however, it is also important to review the use of the resources and their 
application to the particular context. 

The assessor notes that some of the assessments in the learner guide are open to interpretation, 
and learners have queried the apparent ‘repetition’ of certain tasks. After 12 months, the resources 
will be reviewed and appropriate revisions made. However, the lecturer works part time so there 
are real time and resource constraints for adapting assessment materials. 

Availability of qualified staff at the diploma level 

There is also a lack of qualified staff at the diploma level in the childcare centres. Having 
appropriately trained and qualified childcare staff to support the practical placement and 
supervision requirements under the training package is an issue for remote isolated areas. 
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The course lecturer considers that there are no advantages for a childcare centre in having qualified 
assessors on site. Generally, the lecturer assesses with third-party evidence or assessment relies 
solely on third-party evidence. 

Prior to the introduction of the competency-based course and assessment, work placement was 
time based (20 hours spent in a childcare was the unit for work placement) and ‘completion’ of the 
period of time meant the learner had ‘passed’ the module. Workplace assessment requires the 
learners to take more responsibility in managing their learning and assessment. The training 
package is relatively new and places a strong focus on the theoretical approach and the 
underpinning knowledge. 

Support for assessors 

To provide some support for assessors, a moderation group across the region (which meets two or 
three times a year) is being encouraged. 

Strengths of the assessment approach 

<■ The approach to assessment is flexible. 

Written assignments are shorter than assignments in the previous course and they are easier 
and quicker for learners to complete. 

-v- The staggered and smaller assessments used in this course have been more motivating for 
learners than one large assessment event. 

Sr There is immediate feedback from the assessor after the marking, and industry networks are 
well established to support evidence collection. 



Case study 1 0: Going online with industry 

Site lllawarra Institute TAFE NSW 

Qualification Diploma of Extractive Industries Management — MNQ5 01 98 

Unit of Implement, monitor, rectify and report statutory/legal compliance MNQ TL01A 

competency 



Course context or background 

The lllawarra Institute TAFE NSW works with the Institute of Quarrying Australia to deliver the 
Extractive Industries Training Package qualifications (Australian Qualifications Framework Levels 
2—5) to enterprises in the quarrying industry. The delivery and assessment in this training area is 
strongly driven by the peak industry body. Arrangements are also in place for the delivery of 
training for the industry and enterprises in New Zealand. 

Delivery arrangements 

The nature of the industry, locations and employment conditions has created a strong need for 
flexible delivery arrangements for this industry sector. 

The lllawarra Institute has recently developed online learning and assessment resources for eight 
core competencies from the Diploma of Extractive Industries Management. For the module MNQ 
TL01A, learners access online learning resource packages and undertake four learning units. 
Learners complete the qualification through mixed delivery including face-to-face workshops, 
online learning modules, and recognition programs. The registered training organisation has taken 
note of employers’ needs in relation to the delivery mode. When the organisation was setting up 
the program, employers made comments such as: We don’t want it just online, we want the 
socialisation, we want the cross-fertilisation that the face-to-face module will offer’. Facilitators and 
workplace or training provider mentors support the learners throughout the program. 
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learners 



Two types of learner enrol in the qualification: either graduate civil engineers who have moved into 
the quarrying industry later in their career and want to be trained as managers, or longer term 
employees who have worked their way through the system and have well-established workplace 
technical skills, but potential gaps in key competencies. 

Assessment design and methods 

Assessment design was collaborative, with significant industry influence and input. The education 
committees of the Institute of Quarrying provided advice and input into the assessment approach 
and a number of important considerations influenced the approach taken by the registered training 
organisation for this qualification, including: 

-y- guidelines and requirements set out in the Extractive Industries Training Package 

the need to have assessment tools that allow learners to demonstrate competence in the 
context of their workplace 

a process of verification to confirm and assure the assessment process 

the identification at the outset of the application of current levels of knowledge and skill 

mechanisms to provide information to the training provider about the learner’s background 
and needs in order to develop individual development and training plans 

an understanding of the nature of the learner cohort and geographical locations 

-y- the need to meet both the broad-based needs of the industry sector as well as those of 
individual enterprises 

-y- the provision of a variety of assessment pathways in recognition of the anticipated range of 
levels and needs of the learners 

the provision of a high level of formative assessment activity to support the learning process 
^ automated assessment that checks answers and provides immediate feedback. 

Conducting assessment 

Evidence is collected through a variety of assessment methods, depending on the assessment 
pathway negotiated with the registered training organisation. These include: 

on entry, self-assessment of skills and knowledge of all competencies (self-check) 

-v> interactive online short-answer and multiple-choice questions 

-v> online ‘supply’ answers for multiple-choice and true/false questions 

-y- third-party verification of performance 

-y- workplace projects 

portfolio evidence. 

Learner needs and support 

Learners are provided with extensive information about the process and needs of assessment. They 
can also negotiate the assessment pathway and are given regular support in relation to assessment 
through the lecturer and nominated mentor. 
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Orientation session 



A face-to-face orientation program, held at the commencement of all training, provides the 
opportunity for course information to be presented and for the identification of individual learner’s 
strengths and weaknesses. Learners are introduced to: 

-y- the self-check program 

Sr online learning resources 

Sr the terminology of competency-based assessment 
Sr the navigation of the online units 
Sr the requirements for assessment. 

At this stage: 

Particular learning needs such as literacy skills are identified. 

Sr Portfolio evidence is explained. 

^ The types of evidence that can be used for assessment are outlined. 

Learners complete the self-check exercise and identify the evidence they have to support their 
competency so that an individualised learning and assessment plan can be developed. 

Information about the assessment strategy is presented online, including the assessment pathways 
available to learners. The choice of pathway is a negotiated process between the registered training 
organisation and the learner. Familiarisation with the online learning resources and the 
commencement of the assessment process enables learners to function independently back in the 
workplace. 

In response to learner requests for use of the recognition pathway, the organisation has developed 
more explicit guidelines and information about the evidence required for recognition. 

Regular contact with learners 

The training provider has identified regular contact with learners as a critical aspect of support for 
delivery and assessment arrangements online: ‘y ou have to do it [contact learners] regularly ... if 
you don’t do that they can go to sleep on you . . . we learnt the hard way because earlier on we had 
a few who went to sleep’. Currently, a facilitator keeps regular contact with learners. 

Chat rooms and discussion forums are also being planned in future developments to support 
assessment activities. 

Issues for assessment in flexible delivery 

The development, piloting and implementation of the online program during 2000—2001 revealed a 
number of issues and challenges for assessment in flexible delivery. 

Administration costing issues 

Flexibility in assessment and delivery, particularly for the employment of assessors and online 
tutors, raises issues of appropriate payment formulas for the online activity and associated 
assessment. For example, how do you gauge an appropriate rate when workloads vary depending 
on the context of the various competencies within the diploma? This is just one of the challenges 
in regard to the maintenance for the online assessment. 

The registered training organisation is currendy working with large enterprises on the recruitment 
of workplace assessors. Meeting the needs of smaller enterprises in this sector is still to be 
explored, particularly in relation to the workplace verification processes. 
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Authenticity 



The need to be certain that the evidence presented by the learner is his or her own requires a 
verification process. On-site supervisors and workplace assessors verify assessment activities 
undertaken by the learner in the workplace. 

Technology 

Currendy bandwidth issues prevent the use of sophisticated graphical and audio interfaces. The 
design of both learning and assessment has had to take account of the geographical location of 
learners and the speed of loading images and files. One solution to the bandwidth problem has 
been the development of CD-ROM resources. 

Strengths of the assessment approach 

Major strengths of this assessment approach include: 

The strategy supports dexibility in assessment in the provision of a set of assessment pathway 
options for the learner. 

The extractive industries assessment system supports and encourages flexible training and 
assessment arrangements based on partnerships between enterprises and the providers of 
formal structured training programs (<http:/ /www.miningitab.com.au/quarry/quarry/ 
eiagl.htm>). 

A process of recognition of current competence is embedded in the assessment approach, and 
learners can apply for recognition of part or all of the qualification. Industry bodies particularly 
encourage the approach. 

tc The design of assessment tasks supports learners in applying skills and underpinning 

knowledge to their own workplace. The involvement of industry bodies and industry specialists 
in the design of the assessment approach and tools enhances the validity of the assessment in 
this regard. 



Case study 11: Contextualising assessment 

Site TAFE Tasmania Natural Resources 

Qualification Certificate III in Agriculture (Dairy) 

Units of Carry out milk harvesting RUA AG 2528DY 

competency Rear cg|ves R(JA AG 2526 DYA 



Course context or background 

TAFE Tasmania, through its Natural Resources Unit, has delivered training in the dairy industry 
over a long period of time. The original training delivery model was campus based. The impetus 
for a change of delivery arrangements came with the development of the training package that 
specifies workplace assessment as one of the assessment approaches for this qualification, with 
competencies to be assessed in an enterprise context. 

Deliveiy arrangements 

The training is delivered through flexible learning arrangements that feature: 
on-the-job training 

A- an initial face-to-face orientation program 

regional and campus training sessions and workshops (optional) 

-y- on-site visits by lecturers for mentoring and assessment purposes. 
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In theory, the program is all undertaken on the job; however, a delivery model has been developed 
to provide some face-to-face learning. The learning guides provide all the resources for the 
development of the underpinning knowledge for these competencies. However, not all employers 
have the necessary skills to support the delivery of the underpinning knowledge and the lecturers 
believe there are benefits for some learners in the face-to-face mode. 

Learners 

All learners are employed on a farm either as apprentices or trainees. They may be employees or a 
member of a family business. 

Assessment design and methods 

At the outset, the registered training organisation analysed the competency structure of the unit 
using the training package competency standards. 

The selected assessment methods allow the best demonstration of competence through the 
evidence presented. Particular features of the assessment include: 

A tailored, on-site assessment to suit the systems of each enterprise or workplace: an individual 
approach to both the delivery and assessment allows the lecturer to build an information 
profile of the enterprise operation 

A assessor guides, based on the assessment criteria in the training package (original training 
package guidelines were very broad) 

A assessment evidence collected from a range of sources (such as learning guide assessments, 
written evidence, verbal tests, third-party evidence, observation) 

A use of third-party evidence: employers have a role in observations, learner testimonials and sign 
off against the enterprise standards 

A flexibility in assessment: learners negotiate and identify readiness for assessment 

integrated assessment and training planning: as part of the training plan a schedule for 
completion of learning guide sections is developed for each enterprise, taking account of 
seasonal activity and the enterprise’s schedule of events 

A recognition of prior learning. 

Conducting assessment 

Specific assessment methods include: 

A completion of learning guide exercises 
A third-party evidence and testimonials 
A observation by employers and lecturers 
A demonstrations 
A diary entries 

A completion of workplace documents and records. 

For example, for the unit on fencing, the lecturer takes into account: 

A evidence in the diary of the time spent on fencing 

A the supervisor’s testimonial that the learner has completed the work satisfactorily 
A the fence itself 

A learner demonstrations on aspects of fencing 

-v- results of the question-and-answer session on the design and process undertaken. 
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learner needs and support 

General information about assessment is provided at the induction day. As part of the traineeship, 
the lecturer talks to the learners about ongoing assessment requirements and indicates how the 
learner’s log book/ diary is to be completed for the various competencies that are being assessed. 

As a range of assessment methods is used, discussions with the learners include their preferred 
methods or forms of assessment, and learners are informed of the type of evidence expected in an 
assessment. 

The lecturer visits each learner about once a month to monitor progress with the learning guides, 
to conduct assessments, for on-the-job demonstrations and observations and to provide ongoing 
information and mentoring. 

Titeray issues and documented support 

Reasonable adjustment is made for learners with literacy problems, and assessment methods are 
varied. Early identification of such issues results in referral to literacy training; however, this is a 
sensitive area and the majority of learners are not keen to participate in literacy programs. In such 
cases, assessors may rely more heavily on verbal evidence and observation. 

Employer / enterprise involvement 

At the beginning of the program, the employer is involved in the development of a training plan. 
This identifies early in the process how all the competencies are going to be met. If any of the 
competencies cannot be demonstrated at a particular enterprise, the lecturer makes arrangements 
with another workplace, or for a different mentor. The strategies, including contingency plans, are 
identified in the documentation to ensure that both the learner and employer know how the 
process will be conducted. 

Issues for assessment in flexible delivery 

Third-party evidence 

Initially, collection of third-party evidence was a new thing for the employer and there was a lack of 
understanding of the role. This has been addressed through the implementation of an orientation 
program at which expectations are set out and information about the role and resources to support 
supervisors in this role (using checklists) is provided. 

Recording system 

Initially, the recording system posed challenges in terms of fitting with the college-wide 
administration of enrolments and assessment. For example, the college may close off enrolments 
within periods of time, whereas the flexible delivery model has rolling enrolment. 

Assessment database 

An assessment database is being developed from an existing database in the horticulture section 
where delivery arrangements have features in common. This will standardise the forms used for 
assessment and will adapt the content of assessment to meet the needs of a particular assessment 
method. It also has the capacity for questions to be added, providing a handy means for 
moderation in that it will be easy to check and review all the questions being used. 

Strengths of the assessment approach 

The registered training organisation is one of the first to be involved with the training package 
delivery and has had considerable experience in refining its assessment approach. Other strategies 
in place to enhance reliability and build in flexibility include: 
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-v- a strong team approach to delivery and assessment 

-y- clustering agriculture staff, so communication is effective and opportunities for assessment and 
moderation are maximised 

-v- every four to six weeks, meetings of all assessors in the natural resources team across Tasmania 
regular forums on assessment 

A- assessors who have extensive experience in the dairy industry 

well-developed assessors’ tools and extensive information for learners. 

Case study 12: Assessing forensic evidence 

Site Canberra Institute of Technology 

Qualification Diploma of Forensic Investigation (12750 ACT) 

Modules Bloodstain evidence ABD 220 

Forensic physics ABD 204 



Course context or background 

The need for well-trained staff in the area of forensic investigation came to the fore in the period 
of investigation of the death of Azaria Chamberlain in the 1980s. Subsequendy a national course 
was developed (ten years ago) under the Australian Committee on Training and Curriculum, with 
input from the key stakeholders across Australia. 

The Canberra Institute of Technology delivers the Diploma of Forensic Investigation (Crime Scene 
Investigation and Fingerprint Identification) to most police jurisdictions across Australia, having 
been developed originally for the NSW Police Force, the registered training organisation’s first 
customer. It is delivered in a distance mode because of the difficulty of releasing police officers for 
two years of full-time study. Bloodstain evidence is a popular module in the course because it is an 
important feature of courtroom evidence. 

Delivery arrangements 

The diploma is delivered in a distance mode, using learning guide resources. Eight to nine of the 
modules involve a one-to-three-day residential program. The residential sessions are mandatory as 
they provide an opportunity for social interaction and assessment. The training provider has made 
some use of the internet for delivery of chemistry learning activities, and is currendy reviewing and 
developing new activities, but problems with technology capabilities in remote areas have 
prohibited more extensive internet use. 

Discussion groups are maintained for learner interaction. Partnership arrangements are in place, 
with some units being delivered by police training units. 

The learners 

The learners are generally sworn police officers (employees of a state or federal force) who want to 
work in the crime scene unit. Exceptions may be made for areas such as defence forces or other 
units linked to police activity with the permission of the respective police forces. 

Learners undertake this course from all states and territories except Victoria. 

Assessment design and methods 

Assessment activities were designed at the time of the course development, and the course was 
developed with both face-to-face and distance education delivery in mind. 
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The assessment design has been conducted in conjunction with the police, and all assessment 
activities incorporate the forensic emphasis and are related to a forensic context. Consultation with 
and advice from police training units assure the authenticity of assessments. 

Formative assessment is an important part of the design and is used both as a motivational device, 
and as a preparation strategy for the residential component of modules, where a holistic approach 
to assessment is employed: 

We’ve found over the years the more little exercises you can give them, the more confidence 
you can instil in them. When they come in for the residential or the final they say how much 
they appreciate all the build-up to it with the work and exercises. 

The choice of the assessment methods is influenced by the need to make them as ‘real life’ and 
authentic as possible. For example, the style and content of the assessments includes police jargon 
in an attempt to reflect ‘real world’ situations that learners expect and can relate to. 

Assessment review is built into the institute’s practice through course evaluations. This is 
particularly important because of the need to stay in touch with changes in the industry, especially 
changing technology in the forensic science area. 

Recognition of prior learning is also available to learners. 

Conducting assessment 

The assessment methods used for both units include written assignments, practical tests, case 
studies and laboratory reports. These assessment methods allow the integration of practical skills 
and underpinning knowledge. 

An example of an assessment for bloodstain evidence might be as follows: 

In a simulated crime scene during the residential session the police officer: 

-y- carries out occupational health and safety procedures 
-y- makes a drawing of the scene 

y discusses and critiques different options pertaining to evidence at the scene 
y discusses the limitations of the hypotheses 
-y- presents the exercise as a report 

-y- categorises bloodstains at the scene using the correct terminology 
^ predicts the cause of the bloodstains at the scene 
y reconstructs the sequence of events. 

Other assessment tasks include a written assignment and a written test. 

The assessment methods for the forensic physics module are similar. The learning guide for this 
module has a number of scenarios, taking the learner through standard physics. Learners are 
required to: 

y collect, record and interpret measurements 

^ use basic physical concepts and terminology to analyse a vehicle accident scenario 
y make calculations 

y design and carry out simple experiments, including an investigation with a forensic context. 

Formative assessment involves the use of fax to send assessments, as well as the learning guide. 

As part of formative assessment, learners keep a logbook of their activity and progress. During the 
residential session, learners must complete a laboratory report and an investigation report. The 
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assessment is completed with a written text and a case study of a vehicle accident. The class tutor 
makes the final judgement and a high-ranking police officer at the learner’s workplace verifies the 
authenticity of assessments. 

learner needs and support 

The major focus of learner support is on providing information to learners and communication 
strategies that will enhance the assessment approach. 

Once enrolled, all learners receive a learning guide for their nominated module, which includes 
information about assessment including an explanation of the process and the type of activities that 
the learner will undertake. The guide also explains both formative and summative assessment. The 
arrangements for assessment are explained and learners are told what has to be sent to their tutor. 
An additional assessment information sheet accompanying the learning guide provides assessment 
dates/ times and deadlines. 

For high-volume enrolments in these modules, tele-conferences are also organised. 

Communication with learners 

Telephone, fax and email are the major means of communication between learners and the lecturer, 
and are all used extensively. Learners can email assessments, although faxing assignments is 
preferred (to avoid viruses in attachments). 

Distance learners often experience a higher degree of concern about their assignment arriving on 
time. Notification of receipt is given through a fax-back sheet. Individual attention is also given 
over the telephone: ‘. . . we spend a lot more time with individual learners on the phone or by fax 
than you would in a classroom situation for some learners’. 

Issues for assessment in flexible delivery 

Some of the issues relating to assessment which are highlighted in a flexible delivery mode include 
the provision of feedback, the costs associated with assessment, authenticity and security of 
assessments, as well as flexibility of assessment timing. 

Timely and effective feedback is a challenge for all distance-mode deliverers. In the face-to-face 
mode, learners can usually follow up questions about assessment with their lecturer after a session. 
In this program the lecturer, tutors and learners make considerable use of the telephone, which is 
not always satisfactory due to issues of time and cost. 

Learners need to have access to crime scenes, so to maintain course credibility and quality 
assurance, assessments are conducted during residential periods. For final written assignments, 
written verification by a senior sergeant is required. These procedures are time consuming and 
resource intensive because of the varying locations of the learners. Security of assessment is 
maintained via stringent procedures, including setting times with the senior police officer to receive 
the assessment by fax. 

While the assessment plan indicates due dates for tasks, the nature of the learner’s work, where 
they may be called out on a large police operation, requires there to be some flexibility in these 
arrangements. 

Strengths of the assessment approach 

Strengths of the assessment approach include: 

Sr the use of formative assessment 

workplace-related and authentic assessment 
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reliability of the assessment process, including the use of residential sessions for assessment 

the assessment method remains constant but the content of the experiment, the way the case 
study is structured etc., varies from course to course 

the high level of support offered to learners. 

Case study 13: Assessing in remote regions 

Site Tropical North Queensland Institute of TAFE 

Qualification Certificate IV Office Administration (Indigenous) 



Course context or background 

The Aboriginal and Torres Strait Island Indigenous Studies Faculty of the Tropical North 
Queensland Institute of TAFE has designed a Community Management Program for Indigenous 
students whose goal is to manage an Indigenous or non-Indigenous community organisation. The 
program has been designed drawing on the competencies from the national training package BSA 
1997 and is delivered to Aboriginal and Torres Strait Islander students in the area around Cairns, as 
well as to students in remote areas and other parts of Queensland. 

Delivery arrangements 

The learning mode is best described as a mixed-mode delivery pattern comprising options from: 

<■ distance learning guides and resources 
^ residential workshops 
-y- on-the-job learning. 

Learning resources are provided through learning guides incorporating culturally appropriate 
resources. Students can use these resources at home and at their workplace. They include self- 
assessment quizzes at the end of each unit or module. 

The residential component of the program allows the teachers and learners to achieve a range of 
objectives, including the early identification of any literacy or numeracy difficulties that need to be 
addressed. This component of the delivery mode also provides an opportunity to conduct study 
plan interviews with all the participants, arrange and provide study support, and conduct 
equipment and resource-based modules of the program (that is, computer-based and financial 
modules). Students are also encouraged to attempt assessments during the residential sessions 
where support is often more available. 

Residential sessions are also conducted ‘on demand’ on site (community organisation or island) 
where there are sufficient students in one location. 

Learners 

The students enrolled in this program are of Aboriginal or Torres Strait Islander descent, both full- 
time and part-time students, many of whom are already working in community-based 
organisations. 

Assessment design and methods 

Assessment approach 

The training package assessment guidelines (BSA 97) specify that a range of assessment tools be 
used for assessment. They do not specify workplace assessment; however, they recommend that 
simulated workplace tasks may be designed. 
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According to the training provider, the assessment guidelines for this training package are 
necessarily generic because of the diversity of contexts for assessment in this training package: ‘You 
can’t make them test-specific’. 

In general, faculty staff purchase learning guides for these programs and these tend to include 
assessment guides and assessment tools. Some learning resources have been developed by the 
faculty, and so assessment resources have been developed at the same time or by the Product 
Development Unit of the institute, although other suitable resources are sourced from other 
Australian providers. 

Students nominate when they are ready for their assessment and a number of options are available 
including: 

-y- workplace assessment 
written theory assessment 
practical assessment. 

Conducting assessment 

Assessment methods include: 

^ practical tests 
^ project 

written questions 
oral questions 
-v- observation 
<■ workplace assessment. 

Workplace-based assessment is generally undertaken by faculty staff who are qualified certificate IV 
workplace assessors. When students in remote areas indicate they are ready for assessment, the 
institute’s assessors make arrangements to conduct on-the-job, on-site assessments, travelling to 
the student’s workplace or region. 

Adjustments are made in both learning and assessment strategies to meet the particular needs of 
the learners. 

A faculty staff member indicated that adapting to the needs of the learners was an important aspect 
of the assessment process: 

Indigenous people have different ways of learning and performing and we are sensitive to this. 

They are very visual and practical people and we respond to that. 

Learner needs and support 

On first contact with the students, teachers devise a study plan, giving guidelines and indications of 
the approximate hours for each of the courses and the expected timeframes for assessment. This is 
usually conducted during a face-to-face enrolment interview. At the orientation to the course 
session, the course introduction booklet and the learning guides are explained. At this session, 
information about assessment requirements is given. 

New students are invited to attend a one-week residential program at the next scheduled session. 
These residential sessions are held once a month at the various institute campuses or on demand 
when suitable numbers of students are available. During these sessions small tutorial groups are 
conducted and assessment opportunities offered. 
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The college recognises the need and importance of constant and frequent contact with students — 
to monitor learning and assessment progress. To manage this essential contact with students, a 
teacher in the faculty is responsible for contact roles within a particular catchment area. Generally 
the teachers use telephone and fax to maintain this contact. 

We use as many tactics as possible to keep in touch with students. 

The residential sessions are also conducted during the course, particularly to allow learners to 
access the computer equipment and software. While serving a number of purposes for supporting 
assessment, the residential sessions are also an important motivational aspect of the course, 
allowing learners to meet with others in the program, conduct visits to the main campuses of the 
institute and have face-to-face contact with the teaching staff. 

Issues for assessment in flexible delivery 

Time for developing assessment resources 

There are time constraints for the local development of assessment products. Learning guides and 
assessment products therefore have to be able to stand alone as resources for distance learners. 

The resources have to be designed in a way that allows learners to check their own progress. 

Resources for learners 

Many of the regions where the students live and work do not have libraries or satisfactory access to 
the internet. The faculty has to provide all of the resources necessary to support the learning and 
assessment strategies. 

Skilled workplace assessors 

One issue facing the training provider is the lack of skilled workplace assessors in remote areas to 
assist in the collection of evidence — to assess on behalf of the registered training organisation. 
While funding has been sought to train approximately 20 workplace assessors in Cape York and 
Thursday Island in 2002, this does not necessarily solve the problem as the question of payment of 
the assessors still remains. One of the goals of the Community Management Program is to 
encourage course graduates to become tutors or workplace assessors. Competencies from the 
Certificate IV in Assessment and Workplace Training are electives in the Certificate IV 
Administration course. 

Providing support in remote regions 

The faculty staff have developed a number of strategies to support learners in remote regions. 
These include the establishment of local tutorial groups and developing mentoring skills in local 
resource people. 

Recognition of prior learning 

As the recognition of prior learning processes can be seen as very time consuming, students often 
enrol in the program as a refresher course and then choose their assessment times. This allows for 
both review of skills and flexibility in assessment timing. 

Using technology 

Fax is the most extensively used of the communication technologies. It is used successfully for the 
provision of feedback to learners. Video-conferencing between campuses is possible and the 
faculty often sets video-conferencing sessions between its campuses in Cairns and Thursday Island, 
where conference and cam-computer facilities are established. 



86 



Exploring assessment in flexible delivery of VET programs 




Strengths of the assessment approach 

The faculty member discussed the high level of status attached to the awarding of the credential. 
The training provider places a strong emphasis on the course being a nationally accredited 
course/ qualification, so there is a great deal of pride and satisfaction when the students are 
successful. 

The competency-based assessment approach allows participants to show what they are 
competent at doing. 

The ability to respond quickly through the provision of learning resources and the flexibility 
around the timing of assessment is important. 

There are some people who have been out there working for years, say at a council. We enrol 
and send them a learning guide and then a month later they ring up to request an assessment. 

The beauty of flexible learning is just this. 

Adapting the assessment process to meet the needs of the students is seen as a key strength of the 
program. Making the learning and assessment experience relevant and responsive to the needs of 
the individual learner is a strong aspect of the learner management arrangements. 

We consider the individual’s needs, capabilities, interests and preferences. We respond to that 
and we offer on-the-spot support when we can. If parents cannot leave their family then we 
go out to see them. 
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Self-assessment of practice: 
Validity and reliability 



Introduction 

This is the final part of the third stage of the study. 

One of the project’s aims is to explore how the national assessment principles of the Australian 
Recognition Framework (validity, reliability, flexibility and fairness) might be incorporated in the 
assessment practice of flexible delivery providers. To that end, the case-study sites were asked to 
use the self-assessment tool to evaluate features of their assessment design that enhanced the 
achievement of the assessment principles for their program (appendix 3). 

The processes in the VET sector that currendy support the scrutiny of validity and reliability in 
assessment are extremely varied. Review processes are emerging as a result of the implementation 
of the Australian Quality Training Framework. The impact of ANTA-funded reviews and strategies 
to underpin confidence in VET assessment will take some time to become embedded nationally. 
The diversity of current practice and approaches to the validity and reliability of assessment is 
understandable given the range of delivery arrangements and the number of VET providers, as 
discussed earlier in this report. 

Using the self-assessment tool, teachers /trainers are able to identify their strengths, challenges and 
gaps in assessment approaches. Respondents also provide information on specific ways in which 
validity, reliability, flexibility and fairness are enhanced in their assessment. 

This process identified key issues for these providers. 



Design considerations for the tool 

ANTA assessment principles 

The evaluation tool uses the ANTA assessment principles as a benchmark, and incorporates a 
range of descriptive statements developed from current literature (Gillis & Bateman 1999; 
Vocational Education and Assessment Centre 2000; Thomson, Saunders & Foyster 2001; Booth et 
al. 2002). 

Features which enhance the achievement of the assessment principles are described in single 
statements in four sections of the tool: validity, reliability, flexibility and fairness. Using a four- 
point Likert scale, respondents ‘rate’ their achievement of these features. The end points on the 
scale are described as ‘easy to achieve’ and ‘difficult to achieve’. 

Support for a reflective process 

In developing the tool the researchers sought a format that would support practitioners in a 
reflective process. A self-assessment process was chosen because of the perceived value of this 
approach at this stage in the implementation of training package qualifications. 
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The self-evaluation allows for designers of assessment and assessors to: 
reflect on their practice 

<■ increase their awareness of quality-assurance processes in assessment 
-y- identify their strengths and gaps in relation to their current practice 
-v- support continuous improvement in quality assurance. 

Language and terminology relevant to the user’s context 

Exploring validity and reliability issues (within the context of the Australian Recognition 
Framework) with providers is a relatively new area of research for the VET sector. Two related 
studies (Billett et al. 1999; Thomson, Saunders & Foyster 2001) identify some of the issues 
confronting researchers when they attempt to ‘pin down’ validity and reliability with practitioners. 
Inappropriate responses and evidence of lack of understanding of the terms emerge in some of 
their findings. Based on these findings, this study seeks to keep the language and terminology in the 
self-assessment tool as relevant as possible to the user’s context. 

Using the evaluation tool — some comments 

Since the prime objective is to provide a tool for case-study participants to evaluate their own 
assessment approach, comments on the data from responses on the Likert scales must be 
impressionistic. The main benefit for the researcher in reviewing the responses is to determine 
areas of commonality or potential trends in the data, explored below. 

The following ‘trends’ emerge. 

All respondents identified strengths in enhancing their assessment by: 

V the collection of evidence that relates directly to the units of competency or the learning 
outcomes being assessed 

V using different sources of evidence of knowledge and skills that underpin competency 

V clear documentation and communication of the purpose of the assessment and the evidence to 
be collected. 

Fifty per cent of the respondents circled the third or fourth point on the scale (indicating a level of 
difficulty) for the feature: ‘Another person with expertise in the competencies being assessed has 
validated the methods and processes for assessment’. 

Many respondents saw assessment validity primarily enhanced in terms of ‘face’ and ‘content’ 
validity. For example: 

V tasks that are based on or resemble workplace contexts 

V identifying the clear link between the competency to be assessed and the required evidence 

V including stakeholders in the selection of appropriate methods 

V involving content experts in assessment task design 

V review of task match to competency 

V preparing and reviewing detailed task specifications covering knowledge and skills (Gillis & 
Bateman 1999, p.30). 

Respondents reported a higher degree of difficulty for the reliability feature: ‘Validation processes 
are in place with internal and external assessors’. 

The development and use of a self-assessment instrument by participants to evaluate their 
compliance with the national VET assessment principles was a very productive aspect of the 
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project, although very small scale. It allowed participants to engage in an analysis of their own 
experience and to review practice in relation to further improvements. This is an extremely 
important process for building quality in assessment. A range of national initiatives is currently 
addressing validity and consistency in VET assessment practice across Australia. 



Site self-evaluation responses 

The following six tables summarise the responses to questions relating to issues and solutions 
surrounding their own practice in ensuring and enhancing validity, reliability and flexibility 
principles. 

Names that identify specific case-study sites have been removed as the study did not seek to 
specifically evaluate providers’ practice, and for reasons of confidentiality. 

Table 7: Site practice: Ensuring validity 

Provider How validity is enhanced 

A Assessment relates directly to the learning outcomes. 

Assessment is relevant to the workplace, e.g. analysis of a vehicle accident. 

Detailed assessment information is available for learners. 

B Assessment is designed around the relevant training package competency standards. 

C Competence is assessed using the [...] Training Package 

Both formative and summative assessment are carried out in the workplace by authorised workplace 
assessors, to the standards stipulated by the industry competency standards. 

D There is a process of internal moderation by staff involved in the subject area. 

Statewide RTO assessment forums are held to moderate and evaluate assessment. 

Various ITAB and industry representatives have been involved in the assessment meetings. 

E All assessments are carried out by one assessor. 

The RTO assessor uses a range of evidence, including feedback from staff in industry, and makes 
the final judgement as the only person with the qualified assessor status. 

F The training package competency standards and the guidelines are used for assessment. 

The RTO uses a number of assessors and developers across the network. 

G Before the course began, a team of four teachers discussed the assessment items to be used: the 

number, frequency, purpose, measurement of and relationship to the competencies. All four teachers 
were involved in delivery and assessment. 

Before and during each assessment teachers met informally to discuss and validate their 
interpretations of the evidence. 

H A module review form and assessment review process support continuous improvement. The flexible 

learning environment simulates the workplace, helping to improve validity. 

All assessments are negotiable to accommodate specific needs and situations. 

There is explicit assessment of the key competencies. 

I Evidence from the workplace is used — up to three samples against each criteria. 

Overall the approach is one of integrated assessment, however a check-back process is in place to 
ensure sufficient evidence in a range of contexts. 

J A variety of assessment tasks is used to ensure all learning is addressed. 

There are formative assessment tasks that allow the learners to make judgements about their 
learning, i.e. their understanding of the content and their learning. 

Formative assessment activities also add to the summative events not in percentage of pass/fail but 
in the learning requirement to meet summative assessment tasks. 

K Assessment is mostly on site, so evidence is obtained by observation (mostly). 

Some other forms of evidence are gathered where available to supplement the observation (such as 
third-party evidence from the supervisor). 

Note: RTO = registered training organisation 

ITAB = industry training advisory body 
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Table 8: Issues or difficulties for achieving validity and possible solutions 



Provider 


Issues (Q3) 


Suggestions for improvement (Q4) 


A 


As the sole teacher delivering this module it is more difficult 
to get feedback from others. 


Put in place opportunities for professional 
discussion. 


B 


No response 


Confidence in current practice. 


C 


At times the industry competency standards are vague in 
attempts to measure a particular standard in context. 
Companies tend to adopt their own standards in these 
areas. 


Incorporating learner feedback on 
assessment issues into the improvement 
reviews for the course. 


D 


Initial minor difficulties were experienced but largely 
overcome by strategies such as moderation by staff 
participating in the subject area. The RTO division with 
responsibility for our section encourages staff statewide to 
participate in a forum to evaluate assessment procedures. 


Continuing involvement of industry 
representatives through the moderation 
processes mentioned. 


E 


Learners studying from a considerable distance could pose 
a problem. At present we have three remote learners. 1 was 
able to assess them during business related visit. 


1 believe how we conduct assessment at 
this time is sufficient. 


F 




No response 


G 


Competency standards do not really provide a minimum 
‘standard’ or quality of acceptable work, i.e. the 
performance criteria are vague. Some teachers set higher 
standards when using their professional judgement. 


All teachers use the same methodologies 
and criteria, while maintaining validity of 
individual professional judgement. 
Agreement regarding interpretation and 
acceptance of evidence is essential. 


H 


Keeping assessment up-to-date in a fast-moving field of 
electronics and information technology. 


More comprehensive evaluation of 
assessments including longitudinal 
evaluations of the effectiveness of 
graduates in the workplace. 


1 


Levels of performance in the standards are not clearly 
defined. Some workplaces don’t provide appropriate, 
realistic contexts for assessment. Collaborative validation 
with other RTOs. 


No response to this question 


J 


Assessment is judgemental so there may always be 
opportunities for people to interpret evidence differently. 


There is a need to gather evidence from 
as many sources as possible (we already 
try to do this). 


K 


Authenticity of learner responses for summative tasks 
(written assignments) is an issue in this assessment 
method, not only online. The other argument was to only 
allow two to three attempts at the formative assessment 
and to add this towards summative assessment. 


Adult learners need to take account for 
their learning, so it was decided to allow 
multiple attempts at the formative 
assessment tasks. 


L 


Knowledge of the learner’s work and capabilities 


(No response) 




Ensuring audit requirements and competencies and 
assessment requirements in the training package. 


No suggestions 


Note: 


RTO = registered training organisation 
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Table 9: Site evaluation: Ensuring reliability 



Provider How reliability is enhanced 

A Use of a standardised checklist. As the only teacher there is no concern about differences between 

assessors. 

B Assessment is based on the competencies and two supervised assessment pieces. 

A number of learners have undertaken assessment in this program and assessor is satisfied that 
consistency in outcomes is being achieved. 

C The RTO has trained industry-based assessors and audited samples of on-site assessment events in 

order to achieve a level of consistent assessment outcomes. 

An online network between assessors has also been facilitated by the RTO. 

D Established, regular process of internal moderation of procedures by assessing staff. 

Staff liaison with workplace supervisors ensures assessment repeatability. 

E As the sole assessor in the subject one assessor makes final decisions based on evidence. Confident 

in the reliability of the judgement. 

F Checklists are standardised and are used across all assessors. 

Training record books are maintained for each qualification. 

All assessors complete the same Certificate IV in Assessment and Workplace Training. 

G The Toolbox provides the same examples to all learners. 

All learners have the same tasks with the same deadlines. 

Standardised assessment guidelines are provided to all assessors. 

Regular meetings are conducted for discussion and moderation of approaches and learner evidence. 
H Clearly defined performance criteria provided to learners and assessors. 

Assessment processes are guided and supported by module review forms, assessment guidelines 
documents and team discussions. 

Reference models, exemplar assessment tasks and material for assessments are maintained by the 
RTO. 

I Development of standardised list of questions to elicit evidence from participants, used by all 

assessors. 

Development of standard assessment checklists for assessors. 

Use of tape-recorded interviews with candidates. 

Requirement for statuatory declarations for portfolios and verification evidence presented by 
candidates. 

J Third-party reports allow the collection of evidence of competency over time and in different 

situations. 

K Written work with learner and supervisors comments. Practical work — role plays. Observation in the 

work place. 

L Discussion with online facilitators. Model answers supplied in teacher’s guide. 

Double/cross-marking if an assessor is concerned about an assessment event. 

Note: RTO = registered training organisation 
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Table 10: Issues for reliability and suggested changes 



Provider 


Issues (Q3) 


Suggestions for improvement (Q4) 


A 


Often work cannot be marked in one block, i.e. it is very 
common for distance learners to hand work in late. 


No response 


B 


No response 


No response 


C 


Each major quarry company guards their own 
version/interpretation of competency standards as part of 
their commercial edge. 


Occasional interchange (rotation) of 
workplace assessors should create 
cross fertilisation of ideas (also 
encouraged by online network between 
assessors) and aid assessment 
consistency. 


D 


Initially workplace supervisors were unsure of their role in 
assessment and the evidence required. 


Further involvement of workplace 
supervisor in ‘workplace trainer’ courses 
could be beneficial. 


E 


Range of work samples is submitted. 


No response 


F 


Determining minimum standards is always a difficulty. 


Continual review and changes for 
improvement are necessary. 






Check learner cohort against previous 
cohort. 


G 


Part-time staff involved in assessment have less 
opportunity to familiarise themselves with assessment 
issues and processes. 


Even more explicit and rigorous 
evaluation of assessment processes and 
practices. 


H 


It becomes easier to develop appropriate tools as the 
number of assessments carried out increases. 


No response 


1 


As with marking any written assessment there are 
problems with differences/bias between markers; however, 
with good communication this can be overcome. 


Too early to comment. 


J 


No response 


No response 


K 


Unless you are prepared and are able to visit a remote site 
on several occasions, it is difficult to assess someone at 
different times and in different places. Multiple visits to the 
site would become very expensive for the client. 


Make sure you gather as many types of 
evidence as possible. 


L 


Lack of experience or personality clashes between 
supervisor and learner. 


More one-to-one or teacher and group 
(costing/time issues). 
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Table 11: Site evaluation: Ensuring flexibility 



Provider How flexibility is enhanced 



A Some learners receive partial advanced standing. Test can be undertaken at virtually any time. 

B Online assessment in six out of eight pieces allowing flexibility in timing of assessment. Two pieces 

are assessed under supervision and the format of the supervision is also flexible, i.e. workplace, local 
educational facility. 

If learner wished for RPL this can be accommodated. 

C A self-check self-assessment recognition process precedes every competency online. Workplace 

assessors validate the on-the-job assessment. The off-the-job component of assessment is captured 
on the computer. The program manager merges both components to verify competency. 

D Learners with poor literacy and numeracy are accommodated with more oral questioning and 

demonstration assessment. 

Subjects of a seasonal nature may be largely assessed by the workplace supervisor and results 
recorded and verified by oral questioning. 

E We have no choice but to be flexible in order to meet client needs. This fits in with our economic 

environment at present and I strongly believe our flexibility is attractive to clients. 

F The RTO provides access to flexible assessment where needed through negotiations with the 

trainer/assessor. 

RPL/RCC processes are in place for all competencies available in the Training Package qualification. 
G Assessment items are designed to be adaptable to different workplaces or scenarios. 

There is not one right answer. 

Assessment involves learner explanation to supervisor and justification of process followed. 

H All learners are encouraged to 'negotiate' assessment activities to suit individual 

needs/circumstances. 

All assessments provide learners with maximum choice possible in when, where, what and how they 
are done. Our whole flexible learning program is designed and built on the premise of maximising 
choices for learners to best accommodate individual circumstances, e.g. learners choose when they 
are ready to undertake every assessment activity in the course. 

I Portfolios are built up over time through self-paced guides and email/online personal support is 

provided. 

J We have used a wide range of assessment events. 

We have also included a recognition process for each module. 

K Observation is preferable as the basis for assessment, but sometimes it is logistically not possible to 

observe a person working. In these cases you must rely on things such as portfolio and third-party 
evidence from supervisors or clients. 

L Workplace assessment; prior learning and work experience; culturally appropriate and practical. 

M A variety of methods are used. Written work is submitted as well as the on-the-job evidence. 

Note: RPL = recognition of prior learning 

RTO = registered training organisation 
RCC = recognition of current competence 
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Table 12: Issues for flexibility and suggested changes 



Provider Issue 


Suggestion 


A 


All learners need to attend a mandatory residential 
conducted for 15 hours over three days. Some 
assessments are conducted during this period. 


None, as this assessment period allows for the 
‘simulated’ and authentic assessment. 


B 


Flexibility has not been an issue. Learners have 
been happy to complete the assessment as 
stipulated in the program requirements. 


No suggestions. The combination of six online 
assessments and two supervised offline 
assessments has not caused any concerns. 
Learners know the requirements from the start 
of the program and are given clear instructions 
as to procedures. 


C 


No response 


The assessor network is new. As rapport and 
trust are gained progressively through 
experience between the assessors and the 
RTO (credibility building), it may be possible for 
increased assessment flexibility between the 
candidate and the assessor. 


D 


Flexibility does not present notable difficulties. 


The establishment of a register to identify 
obstacles experienced statewide may lead to 
difficulties being identified, and collaborative 
activities for solutions and problem-solving. 


E 


Provide flexibility in assessment while meeting 
operational demands of the course and 
organisation. 


No response 


E 


The main difficulty is knowing the weightings that 
different workplaces would place on elements, and 
performance criteria for competencies. 


More links with workplaces and workplace 
assessors, although the range of variability 
across workplaces may be great. 


F 


Restrictions imposed by more traditional (less 
flexible) ‘systems’ e.g. the requirement to assess 
and result every module at the end of the year. Our 
flexible learning program imposes NO start or stop 
dates on our learners, but the system requires us to 
give results for all modules, then roll over all 
modules that learners will continue with next year. 


No response 


G 


Some clients may become too complacent, given 
our level of flexibility. 


No response 


H 


The difficulty is in being able to make a judgement 
of competency when you haven’t seen the person 
demonstrate the skill. You must have faith in the 
other forms of evidence gathered. 


If the person has maintained a log of jobs 
completed and this is verified by a supervisor, it 
would make the assessment process easier. 


Note: 


RTO = registered training organisation 
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Conclusions 



The Australian VET sector is diverse and complex. This small-scale pilot study into assessment 
practice in flexible delivery was undertaken in that wider context. Clients of the VET sector are as 
varied as the providers, studying full or part time, employed, highly skilled or seeking entry to the 
workforce. This study revealed how difficult it is to describe activities in vocational education and 
training as if it were one uniform sector. Also, as flexible delivery models emerge or merge, a 
‘borderless’ VET marketplace develops. 

The growth in the use of information communication technologies will impact on VET flexible 
delivery over the next decade. This suggests both challenges and opportunities for practitioners in a 
competency-based training and assessment system. Foremost among these is the design of 
assessment approaches that can achieve a balance between the needs of the individual, industry and 
the requirements of the national VET system. 



Implications for assessment practice in flexible delivery 

Models of good practice 

The study did not set out specifically to identify good practice; however, those practitioners in this 
study who are ‘testing the boundaries’ in assessment in flexible delivery provide useful models and 
strategies for others in the field. They highlight four critical issues for assessment in flexible 
delivery: 

V Learners need to be ready to undertake more individualised flexible learning arrangements. 

V Well-designed, developed and resourced support strategies for learners — before, during and 
after assessment — are absolutely necessary. 

V Systematic formative assessment and feedback processes that add value to the learning 
experience need to be in place. 

V The learner’s context is an important consideration in the assessment design process for 
flexible delivery. 



Flexible and integrated assessment processes 

Flexible delivery, whether it is called ‘mixed-mode’, ‘distance’, ‘open’, ‘blended’, ‘hybrid’, 
‘e-learning’, ‘online’ or ‘work-based’ learning, implies a wider range of options for VET learners. 
Inherently it also implies a stronger emphasis on catering for individual needs. The implications for 
assessment and resources have yet to be fully realised. As learning arrangements diversify, 
considerable planning and thought will need to go into the assessment design stage to ensure that 
assessment is integrated, valid and reliable. Design and management systems need to be in place to 
support the integration of learning and assessment strategies involving a range of delivery modes. 
For example, ensuring that assessment conducted on the job is well integrated with other 
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assessment components in a module or course conducted online, or in a learning centre, is a critical 
assessment design consideration. 



Building in self-assessment 

The planning and development of well-designed self-assessment tools for a range of assessment 
purposes (both diagnostic and formative) in flexible delivery is an important aspect of VET 
assessment that needs greater attention. Both the literature and case-study informants highlight the 
need to provide opportunities for assessment practice, reflection and self-assessment activities. 

This must be built into the design of the program from the outset. In view of the importance of 
recognition of prior learning or current competence as an assessment pathway, the development of 
self-assessment tools for use in this process would be beneficial to both candidates and assessors. 

There are strong pedagogical reasons for a greater involvement of learners in the assessment 
process. Self-assessment is an integral part of learner-centred approaches to instruction that aim to 
encourage the active participation of the learner in each stage of the teaching or learning approach, 
including assessment. The benefits derived from using self-assessment include helping learners to 
become skilled judges of their own strengths and weaknesses; self-assessment also assists in the 
process of setting realistic goals. Learners need to be trained to use self-assessment techniques. 



Working within existing constraints 

While teachers and training organisations strive for flexibility in both delivery and assessment, they 
operate within a range of institutional requirements and operational constraints. Choices in the 
sequence of modules within a course, open enrolment periods, multiple input to assessment by a 
range of personnel and tracking learner progress in a flexible learning pattern are important 
considerations for flexible delivery. For example, open enrolments and flexibility in the choice of 
course module sequence may meet the needs of some learners; however, establishing cohorts of 
learners engaged in similar activities at the same time may be more pedagogically sound. 

Practical factors such as these impact both on assessment processes and teacher workloads, and 
highlight the need for individual assessment plans that are logistically feasible in the light of 
available resources. 



Security and authenticity of student performance 

The issue that assessment evidence provided by learners is authentic is a concern across all 
educational sectors. The concerns appear to have grown with the development of more flexible 
learning and assessment arrangements — particularly online learning. It is suggested that 
opportunities for cheating and plagiarism are greater in this mode. While it is imperative that 
training institutions ensure confidence in assessment outcomes, it is also important that assessment 
practices do not inhibit assessment flexibility. Supervision of assessment is only one of a range of 
assessment strategies to ensure authenticity. Well-designed assessment tasks, incorporating a range 
of continuous assessment components and regular contact with learners, are some of the strategies 
recommended by providers to address the concerns. 

Using third-party evidence 

The use of third-party evidence has significant implications for teachers and assessors, and their 
organisations. Typically, third-party evidence is sought from current or previous employers or 
co-workers. As employers, willingly or not, become involved in a competency-based assessment 
process, the need for an educative process for all stakeholders is apparent. Without this process 
and the systems for quality assurance of evidence in place, confidence in the assessment outcomes 
is reduced. The lack of qualified workplace assessors for remote and rural areas is another issue of 
concern. 
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The future 



In summary, the challenges that have been identified by this study will need to be addressed as 
flexible delivery modes continue to grow. Designers, practitioners and managers will all need to 
consider assessment issues in the planning and development stages of flexible learning design. 
Some of the examples of practice in this report highlight strategies taken by registered training 
organisations to address some of these challenges and may provide some guidance for others. 
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Appendix 1: Provider survey 



FAX BACK SURVEY 

NREC Project: How flexible is assessment in the flexible delivery 

of VET Programs? 

To: Patricia Hyde Fax No: 02 9448 4560 

Email: Patricia.Hyde@det.nsw.edu.au Phone: 02 9448 4553 

A letter was sent to your organisation explaining the purpose and background of the 
survey. Please contact Patricia Hyde if you would like further information. 



RESPONDENT DETAILS 

1. Name: 

2. Contact Phone No: 

3. Email: 

PROVIDER DETAILS 

4. Name of Registered Training 

Organisation: 

5. State/Territory (Please tick the State or Territory of Primary Registration) 

□ ACT □ NSW □ NT □ OLD □ SA □ TAS □ VIC □ WA 

6. Type of Registered Training Organisation (RTO): (Please Tick) 

□ TAFE L Private □ Community L Enterprise □ Other 
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COURSE OR QUALIFICATION DETAILS 
Please Note 

To complete this survey please select a module/unit of competency or related group of 
modules/units of competency from a Training Package qualification or accredited course 
delivered in a flexible mode. We are particularly interested in modules/units of 
competency where a range of assessment methods has been used to collect evidence. 



7. Name and AQF level or T raining 
Package qualification or accredited course. 

(Please also give the National (NTIS) code 
for the qualification/course if available) 




8. Name of Training Package 




9. If an accredited course, does this lead to □ Yes 
a Training Package qualification? 


□ No 



10. Name and AQF level of units/ modules 
selected for the survey 

(Please give the National (NTIS) Code/s for 
these modules/units if you have them) 



LEARNING/DELIVERY ARRANGEMENTS 

1 1 a. What are the learning/delivery arrangements for the qualification or course you 
have selected? (Please tick one of the following and briefly describe these 
arrangements) 

□ Distance using print and/or other media resources 

□ Wholly online delivery 

L Mixed delivery mode (using a combination of modes, eg online, face to face) 

□ Open or Learning Centre Other 



NB All questions from here apply to the modules/units selected for the survey 
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1 1 b. What are the learning/delivery arrangements for the module/units of competency 
you have selected? (Please tick one of the following and briefly describe these 
arrangements) 

□ Distance using print and/or other media resources □ Wholly online delivery 
L Mixed delivery mode (using a combination of modes, e.g. online, face to face) 

Open or Learning Centre Other 



12. Where are learners located for their study? (Tick as many as apply) 

L at a Learning Centre in a college or workplace □ Workplace 

L Home Information not available 

13. For the modules/units selected is there a requirement for learners to be currently 

working in the industry? □ Yes □ No 

If Yes, please provide details 



14. Is any assessment conducted in the workplace? L Yes No 

If Yes, please provide details 



ASSESSMENT REQUIREMENTS 

15. Were any of the Training Package Assessment Guidelines used for assessment 
planning for the modules/units? L Yes No 

16. Were any other assessment guidelines or resources used? L Yes No 
If yes, please give details 



17. Are there any special assessment requirements associated with the 
modules/units? (eg assessment must be conducted in a workplace?) 

L Yes No 
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If yes, please describe briefly. 



If yes, who determines these requirements? (eg the Training Package, RTO) 



ASSESSMENT DESIGN, STRATEGIES AND METHODS 

18. At what stage and by whom were the assessment strategies determined for the 
modules/units (ie approach to assessment and evidence gathering) 

L By Training Package developers during the development of Learning 
Resources for the Training Package 

By RTO course designers at the course design stage 

□ By Individual assessors/teachers at the implementation stage 

□ Other 



19. At what point and by whom were assessment methods determined for the 
module/units? (ie particular techniques or procedures used to gather evidence) 

L By Training Package developers during the development of Learning 
Resources for the Training Package 

By RTO course designers at the design stage 

□ By Individual assessors/teachers at the implementation stage 

Other (Please specify) 



20. What factors or considerations affected the choice of the assessment methods for 
the module/units in flexible delivery mode? 
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21 . Please describe up to five assessment methods used to collect evidence for 
achievement of competence? (ie for summative or final evidence) 

a) 



b) 



c) 



d) 



e) 



22. What assessment methods are used to support learners and provide evidence of 
learner progress during training? (ie for formative assessment) 

a) 



b) 



c) 



d) 
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e) 



CONDUCTING ASSESSMENT 

23. Who is involved in making final judgements about the learner’s competence for the 
module/units selected? (Tick more than one if applicable) 

□ teacher/trainer/instructor □ qualified assessor in the workplace 

□ workplace supervisor or manager □ other (eg industry expert, peers) 

(Please specify) 



GENERAL 

24. What issues are there for competency based assessment in flexible 
delivery modes? 



25. How have these issues been addressed in the modules/units you have selected? 



26. Any other comments about assessment and flexible delivery? 



YOUR ROLE 

27. Which of the following best describes your role in the training and assessment for 
this course or module? (Please tick and comment) 

Course developer Teacher/Tutor/Facilitator Assessor 

L Multiple Roles Other (Please specify) 
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FURTHER CONTACT FOR THE PROJECT 



28. Would you/your organisation be prepared to participate further to assist this 
project? This would involve an interview with a designer or assessor of up to 45 
minutes each interview (via phone, email or face to face). 

L Yes No 

29. Who should we contact for permission to: 

• interview a developer and an assessor 

• cite resources/materials developed by the organisation 



30. Who should we contact for further information about assessment design of 
this course? 



31 . Who should we contact for further information about the delivery and assessment 
of this course? 



THANK YOU VERY MUCH FOR GIVING YOUR TIME TO PARTICIPATE IN THIS SURVEY. 

Patricia Hyde 
Project Officer 

Vocational Education and Assessment Centre 
Phone: 02 9448 4553 
Fax: 02 9448 4560 
Email: patricia.hyde@det.nsw.edu 
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Appendix 2: Provider matrix sheet 1 (1—10) 



Provider 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


AQF level 
qualification 


Cert. 2 


Cert. 2 


Cert. 2 


Cert. 2 


Cert. 2 


Cert. 2 


Cert. 2 


Cert. 3 


Cert. 3 


Cert. 3 


Training 
package 
stream 
or course 


Business 

Admin. 


Information 

Technology 


Food 

Processing 

FDF98 


Information 
Technology 
ICA 20199 


Textile 

Production 

(LMT00) 


Agriculture 

(Dairy) 

RUA98 


Retail 
Operations 
WRR 20197 


Education 
(Office support) 
(Course) 


Boat Building 
(Course) 


Tourism 
International 
Retail THT 98 


Units or 
module 


NOS 124 
NOS 216 


ICAIT014B-17B 


Safety 

FDF 

COROHS2A 


ICAITU006B/12 
Operate 
Computer 
Packages 
Design docs. 


LMTPRTX01A 
Operate 
machine for 
production 


Rear calf 
Milk harvesting 


Combined 
Using Toolbox 


FIN 301-05 
NOS 124 


Combined 


THT TCOOIO 


Industry 

sector 


Business 

Administration. 


Information 

technology 


Food processing 


Information 

technology 


Textile clothing 
& footwear 


Agriculture 


Retail 


Education 

administration 


Marine 


Tourism 


Provider 

status 


Public 


Community 


Public 


Public 


Private 


Public 


Public 


Public 


Public 


Public 


State/territory 


NSW 


OLD 


WA 


VIC 


VIC 


TAS 


WA 


VIC 


NSW 


NT 


Delivery mode 


Online 

Module from a 

mixed-mode 

course 


Mixed 

Open learning 
centre 


Workplace 
On & off the job 


Mixed-mode 
Learning centre 
Workplace 


Mixed 
Workplace 
Onsite with print 


Mixed mode 
Distance 


Distance 
Online course 


Online 

Module from a 

mixed-mode 

course 


Learning centre 
with learning 
guides & on-site 
support 


Distance (print) 
and online 


Features of 
assessment 
approach 


Final 

assessment in 
workplace or 
classroom 
Observation 
Holistic 


Pre and post- 
testing 
Immediate 
feedback 
Automatic 
assessment 
system 
management 


Workplace 

assessment 


Workplace 
assess, for 
trainees 
Supervision 
arrangements 
for distance 


Workplace 
based (or 
simulated) 

Use of 
workplace 
technical expert 


Workplace 
assessment and 
other evidence 
for trainees 
Contextualised 
assessment 


Assessment via 
email 

Workplace 

assessment 


Module: 

Combined, 

supervised and 

online 

Teacher 

assesses 

evidence 


Industry input 
Assessment at 
learning centre 
Continuous 
assessment 
Learner choice 


Workplace 
assessment for 
trainees 


Assessment 

summative 


Case study 
Questioning - 
short answer 
Practical project 
Skill sample 


Online challenge 

tests 

Project 

Skill 

Demonstration 


Observation on 
the job 
Interview 
Supervisor 
report 

Demonstration 
Role play 
Oral questions 


Observation 
projects 
Case study 
Tests (under 
supervised 
conditions) 
Journal 
Log of duties 


Direct 

observation of 

assessee 

Witness 

testimony 

Questioning 

Work sample 

Training records 


Work diary 

signed by 

supervisor 

Written 

assignments 

Projects 

3 rd party 

evidence 

Questioning 


Portfolio 

3 rd party 

evidence 

Signoff by 

supervisor 

Questioning 

(telephone) 

Written 

assignment 


2/8 formal 
supervised 
assessment 
events 
Skill sample 


Sea trials 
Questioning — 
written and oral 
Practical project 
Module test 


Short-answer 
questions 
Group poster 
Oral 

presentation 

On-site 

questioning 




Provider matrix sheet 2 (11—20) 



Provider 


11 


12 


13 


14 


15 


16 


17 


18 


19 


20 


AQF level 


Cert. 3 


Cert. 3 


Cert. 3 


Cert. 3 


Cert. 3 


Cert. 3 


Cert. 3/4 


Cert. 4 


Cert. 4 


Cert. 4 


Qualification 


Drilling 


Education 


Engineering 


Local 


Child Care 


Aquaculture 


Office Admin. 


Airline 


Small Business 


Transport & 


Training 

package 

stream 


(Mineral Ex.) 


support 

(course) 


fabrication 
MEM 30398 


Government 

Operations 


Community 

Services 




(Indigenous) 


Operations 
(Leadership & 
management) 


Management 


Distribution (Rail 
Services) 


Units or 
module 


Combined 


Combined 


Combined 


Combined 


CHCCN2A 
Provide physical 
care 


SFIAQUA308A 
Maintain water 
quality & 
environ, 
monitoring 


Combined 


People Manager 
EEO 

OH&S & other 
modules from 
ZQF00 


Dealing with 

Conflict 

NCS005 


Drive train to 
agreed 
operational 
standards 


Industry sector 


Drilling 


Education 


Metal & 
Engineering 


Local 

government 


Community 

Services 


Seafood 


Business 

Administration 


Transport 

Airline 

Operations 


Business 

Administration 


Transport 


Provider status 


Private 


Public 


Public 


Public 


Public 


Public 


Public 


Enterprise 


Public 


Enterprise 


State/territory 


NSW 


VIC 


OLD 


NSW 


WA 


SA 


OLD 


OLD 


OLD 


NSW 


Delivery 


Distance 


Mixed 


Open learning 


Mixed 


Open & learning 


Mixed 


Mixed 


Online 


Mixed 


Mixed 


mode 


Mixed 

Workplace 


Print, online, 
workshop 


centre 

Workplace 


Learning centre 
Home 


centre 

Distance 


Distance 


Distance and 
residential 


Workplace 


Online 


Online 
In classroom 
In simulator 
On the job 


Features of 


Workplace 


Need to be in 


Workplace 


Workplace 


Have to be 


Workplace or 


Workplace 


Workplace- 




On-the-job 


assessment 


assessment 


contact with 


Assessment for 


assessment 


assessed in 


simulated 


assessment 


related 




assess, required 


approach 


On the job 
RPL 


educational 

setting 


trainees 

Final 

assessment in 
workplace 


Range of 
evidence 


licensed 
premises. Uses 
self-assessment 


Negotiation 
between 
learners & 
workplace 


Focus on the 

individual: 

negotiation 


Simulated 

environment 




Evidence 
collected from 
range of sources 


Assessment 


Assessor 


Activities 


Observation 


Written 


Written tests 


Methods 


Assessment 


Self-assessment 


Pre-test 


Trainee 


formative 


feedback 
Learning guide 
exercises 
Observation 


undertaken in 
learner guide 
Email contact 
Chat room 


Assessor oral 
questioning and 
feedback 
Log book 
Practice 


Individual or 
group (using real 
job), 

On-the-job 
challenge test, 
RPL, oral written 


On-the-job diary 


selected on the 
basis of 

individual needs 


tasks in learning 
guide 

T utor contact 
through 
phone/fax 
Residential 


Computer- 

mediated 

quizzes 


Interactive 
feedback from 
tests and 
quizzes 


assessment 
record book 
used to record 
evidence 


Assessment 


Holistic 


Written reports 


Log book/job 


Workplace 


Written 


Demonstration 


Observation 


Written reports 


Postings on 


Online quizzes 


summative 


Observation 


Case study 


diary 


supervisor 


assessment 


Written 


Skill 


Workplace 


discussion 


Classroom- 




Question 


Practical product 


Written tests 


evidence 


Logbook signed 


assignments 


Demonstration 


project 


board 


based 




(checklist) 


Resource folder 


Oral questions 


Learner training 


by training staff 


& quizzes 


Supervisor 


Case study 


Emails 


observation 




3 rd party 
evidence 


Quiz, multiple- 
choice 

questions, short- 
answer 


Practical 

demonstration 

Simulation 


log 

Feedback 


Portfolio 

evidence 

Observation 

Interview 


Practical 

exam 


report 


Notice board 
participation 


Role play 


Classroom- 
based tests 
On-job practical 
Demos 



Note: ADITC = Australian Drilling Industry Training Committee 
RPL = recognition of prior learning 
EEO = equal employment opportunity 
OH&S = occupational health and safety 




Provider matrix sheet 3 (21—30) 



Provider 


21 


22 


23 


24 


25 


26 


27 


28 


29 


30 


AQF level 


Cert. 4 


Cert. 4 


Cert. 4 


Cert. 4 


Cert. 4 


5 


5 


5 


5 


6 


Qualification 
Training 
package stream 


Certificate 

Business 

Admin. 


Information 

Technology 

(MultiMedia) 


Fiorticulture 

Production 


Information 

Technology 

(Systems 

Analysis) 


Frontline 

Management 


Diploma of 
Forensic 
Investigation 
(Course) 


Diploma of 
Extractive 
Industries 
Management 


Diploma in 

Frontline 

Management 


Diploma of 
Engineering 
(Computer 
Systems) 


Grad. Cert in 

Wound 

Management 


Units or module 


Meetings-Org. 
Meet client 
needs 
Plan/record 
meeting 


ICPMM65DA 

ICAITB075A 


RUH HRT412 
Develop an IPM 
plan 

RUH HRT 435 
Cost a project 


Combined 


Combined 


Bloodstain 
evidence 
Forensic Physics 


MNQ TL01 
Statutory Legal 
Compliance 


Combined 


Soldering 
& Key 

Competency 

areas 




Industry sector 


Administration 

services 


Information 

technology 


Horticulture 


Information 

technology 


Cross-sectoral 


Public safety 


Extractive 

industries 


Cross-sectoral 


Electronics & 

information 

technology 


Health 


Provider status 


Public 


Private 


Public 


Public 


Public 


Public 


Public 


Private 


Public 


Public 


State/territory 


NSW 


QLD 


Victoria 


ACT 


NSW 


ACT 


NSW 


Tasmania 


SA 


NSW 


Delivery mode 


Mixed 


Online 


Mixed 

Online (core 
units not online) 


Mixed with 
online using 
Tool Box 


Mixed 

Online 


Mixed 

Distant 


Mixed 

Online 


Initial self- 

assessment 

Portfolio 


Mixed 

Learning centre 
Online 


Online 


Features of 
assessment 
approach 


Negotiated 
workplace 
assessment 
available as 
alternative to 
classroom 
assessment 
Timed 

assessment as 
required by the 
training package 


Mandatory 
participation in 
the RTO web- 
based 
discussion 
board 


Customise 

assessment for 

clients 

RPL for core 

Workplace 

assessment 


Integrated 

assessment 

Project-based 

Simulation 

experience 


Workplace-based 
Context-driven & 
collaborative 


Simulated 
environments for 
learning and 
assessment of 
practical & 
underpinning 
knowledge 


Workplace and 

industry 

partnerships 

RPL, self- 

assessment 

Workplace 

assessment 

Computer 

assessment 


Collaborative 
and context- 
driven 
Range of 
evidence 
Multi stake- 
holder 
environment 


Negotiated with 

learners 

Integrated key 

competency 

assessment 

Computer- 

mediated tests 


Integrated 
assessment of 
knowledge & 
skills through 
online case 
study scenario 
Learning & 
assessment 
interwoven 
RPL available 


Assessment 

formative 


Practical test 
online 
Project — 
research 
Portfolio 


Creation of web 
pages, directory 
& file/folder for 
site, navigation 
& installation of 
3 rd party 
software 


Assignments 

submitted 

Self-assess 

activities 

Quizzes 

(true/false) & 

short answer 


Submit project 

deliverables 

Products 

Observation 

Log sheet for 

self- 

assessment 


Workplace visits 
Observation 
Interview 
Unit workbook 
exercises 
Case studies 
Draft portfolios 


Learning guide 
activities 
Laboratory log 
book 


Self-check 
computer- 
mediated skill 
assessment 


Face-to-face 

interviews 

Phone 

interviews 

Online 

discussion in 

forums 


Self-check 
computer-based 
assessments 
Tutor feedback 
Check points in 
practical activities 
Computer 
tracking of 
learning 


Worksheets 

Role-play online 

Multiple-choice 

questions 

Practical 

exercises 

Chat room 


Assessment 

summative 


Self-assess 
multiple-choice 
questions, short 
answer, 
matching 
Participation in 
online chats 
Case studies 


Assessment 
feedback sheets 
Emails for 
specific 
questions 


Written 

assignments 

Self-assessment 


Web CT 
provides 
results 
Feedback on 
all tasks — see 
survey 


Portfolios of 

workplace 

performance 

(reports, 

products, 

projects) 

statements, 

Observation 

Simulation 


Laboratory report 
Case study 
Simulation 
Written test 
Project 
Log book 
Investigation 
Report 


Workplace 

project 

Written report 

Short-answer, 

multiple-choice 

questions 

Case study 

Portfolio 


Oral & verbal 
questions 
through phone 
& email & 
interview 
Portfolio desk 
assessment 


Computer-based 
questions 
Practical project 
Demonstration of 
skills 

Written reports 
Oral presentation 


Multiple-choice 

questions 

Short-answer 

questions 

Essay 




Appendix 3: Self-assessment tool 



EVALUATING YOUR ASSESSMENT: ENHANCING VALIDITY, RELIABILITY, 
FLEXIBILITY AND FAIRNESS IN ASSESSMENT 



Dear .... 

Thank you for agreeing to participate in evaluating the assessment approach used in your VET 
course. 

We are asking you to evaluate your assessment approach to provide us with greater insight into 
assessment design issues, particularly for flexible delivery in VET. However reviewing 
assessment is also good practice as it can help you identify any areas that may improve the 
quality of assessment. 

To assist you to review your assessment approach we have put together a list of features for 
each of the assessment principles. We have drawn up this list from a range of existing research 
and policy documents. Our intention is for the lists to serve as prompts for you to reflect on your 
own context. 

When you consider the assessment principles in relation to your own context you may find that 
you have paid more attention to some than others, or some are easier to achieve than others. 
This is to be expected given the wide range of assessment contexts surrounding assessment 
practices in VET. 

What we would like you to do. 

❖ Read the short paragraphs about validity, reliability, flexibility and fairness on the 
following pages and consider these in light of the assessment approaches for the 
course you have nominated for this project. 

❖ Read the list of features for each of the assessment principles and then use the scale to 
reflect your experience for the course. 

❖ Briefly outline examples of how your assessment design has addressed the principles. 
(The list of features we have described may be useful as a prompt.) You may also find 
that some of your examples cover more than one of the principles - this is to be 
expected given the nature of integrated and holistic assessments. 

❖ Comment on any difficulties or issues you encountered in your assessment approach in 
flexible delivery 

❖ Suggest any changes that could be made to improve your assessment procedures. 

❖ Outline any differences you have found in the assessment approach if you have 
delivered this course in a traditional classroom based delivery model. 

❖ FAX BACK to 

(* These principles are derived from the Australian Recognition Framework) 

Many Thanks 
Patricia Hyde 
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VALIDITY 



Refers to the use and interpretation of evidence collected. A valid assessment assesses what it 
claims to assess; evidence collected is relevant to the activity and demonstrates that the 
performance criteria have been met. Validity refers to the soundness of the interpretations and 
uses you make of your assessment results. 

Assessment is improved when the designer of the assessment has considered features which 
enhance validity. 

1 . Read the following list of features and then circle one of the 4 points on the scale which best 

approximates your experience in designing and conducting assessment for this course. 

Some features enhancing valid assessment Easy to Difficult 

achieve to achieve 

The evidence relates directly to the units of 
competency, or learning outcomes, being 

assessed 1 ! | 

More than one task and source of evidence is 
used as the basis for judgement, with evidence 
drawn from a variety of performances over time 

where practical | j i 

Different sources of evidence of knowledge and 
skills that underpin competency are used in the 

assessment | ! ! 

The instrument assesses the candidate’s ability 
to meet the level of performance required by the 

unit(s) of competency I | | 

Assessment tasks are based on realistic 
workplace activities and contexts 

The assessment tasks have been designed to 
allow integrated (holistic) ** assessment of 

knowledge, skills and attitudes | \ | 

There is clear documentation and 
communication of the purpose of the 

assessment and the evidence to be collected | | | 

There is clear documentation and 
communication about the way the evidence will 

be interpreted and judged | j j 

Another person with expertise in the 
competencies being assessed has validated the 

methods and processes for assessment I I I 

"‘Integrated assessment is an approach to assessment that covers multiple elements and /or units from relevant 
competency standards. The integrated approach combines knowledge, technical skills, problem solving and 
demonstration of attitudes and ethics into the assessment tasks, (p. 1 36 Glossary of Terms, Training Package for 
Assessment and Workplace Training BSZ98, ANTA 1998) 

2. Briefly outline some of the ways you have used to ensure valid assessment in this course. 

3. Comment on any difficulties or issues you have identified in achieving validity. 

4. Suggest any changes you feel would improve the validity of your assessment approach. 
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RELIABILITY 



Refers to the extent to which an assessment can provide repeatable outcomes (consistency) 
for candidates of equal competence at different times and different places 

Assessment is improved when the assessment designer has considered features which 
enhance reliability. 

1. Read the following list of features and then circle one of the 4 points on the scale which best 
approximates your experience in designing and/or conducting assessment for this course. 

Some features enhancing reliability Easy to Difficult to 

achieve achieve 



Critical elements of the competency 
standards are identified and sampling is 
used to ensure the most important aspects 

are assessed | j j 

Assessment exemplars and standardised 

checklists are available for assessors | j j 

Clear assessor guidelines are available to 
ensure that assessors make consistent 
decisions over time and with different 

candidates | | | 

Guides for observing and recording evidence 

are based on units of competency | j j 

Agreed procedures are in place for multiple 
assessors involved in conducting parallel 

assessment events | j j 

Consistent instructions to candidates and 
procedures for undertaking assessment are 

available to all assessors | j j 

Where work samples are to be used as 
evidence, candidates receive specific 
guidelines on requirements, including 
information about ensuring authenticity and 

currency of the evidence | j j 

Where competencies are going to be 
assessed in different situations, the levels of 

difficulty are comparable | j j 

Validation processes are in place with 

internal and external assessors | | | 

2. Briefly outline some of the ways you have used to ensure valid assessment in this course. 

3. Comment on any difficulties or issues you have identified in achieving reliability. 

4. Suggest any changes you feel would improve the reliability of your assessment approach. 
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FLEXIBILITY 



Refers to an assessment approach and methods which are responsive and appropriate to a 
range of delivery modes, sites of delivery and the needs of learners. Variability in the elements 
of timing, location and methods used to meet learner needs has generally been accepted as a 
basic feature of flexible assessment. 

Assessment is improved when the assessment designer has considered features which 
enhance flexibility. 

1 . Read the following list of features and then circle one of the 4 points on the scale which best 
approximates your experience in designing and/or conducting assessment for 
this course. 

Some features enhancing flexibility Easy to Difficult to 

achieve achieve 



The assessment approach can be 
adapted to meet the needs of all 

candidates and workplaces | | | 

Assessment can be negotiated and 
agreed between the assessor and the 

candidate | j | 

Candidates can have previous 

experience or expertise recognised | j | 

The assessment strategy adequately 
covers both the on- and off-the-job 

components of the training | [ I 

2. Briefly outline some of the ways you have used to ensure flexible assessment in 
this course. 

3. Comment on any difficulties or issues you have identified in achieving flexibility. 

4. Suggest any changes you feel would improve flexibility in your assessment approach. 
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FAIRNESS 



Refers to assessment design practice which ensures that assessment processes and methods 
are equitable to all groups of learners. 

Assessment is improved when the assessment designer has considered features which 
enhance fairness. 

1. Read the following list of features and then circle one of the 4 points on the scale which best 
approximates your experience in designing and/or conducting assessment for 
this course. 

Some features enhancing fairness Easy to Difficult to 

achieve achieve 



Candidates are given clear and timely 
information on assessment including 
assessment methods, procedures, the 
criteria against which they will be 
assessed and when and how they will 
receive feedback 



Candidates are included in discussions on 
the choice of assessment methods and 
timing and are made aware of their 
responsibilities with regard to assessment 



The assessment approach chosen caters 
for the language, literacy and numeracy 
needs of all candidates 



The special geographic, financial or social 
needs of candidates are considered in the 
development and conduct of the 
assessment 



Procedures support the use of allowable 
adjustment by assessors, while 
maintaining the integrity of the 
assessment outcomes 



Opportunities for feedback and review of 
all aspects of assessment are provided to 
candidates 



There are clearly documented 
mechanisms for appeal against 
assessment processes and decisions and 
these are provided to candidates prior to 

assessment III | 

2. Briefly outline some of the ways you have used to ensure fair assessment in this course. 

3. Comment on any difficulties or issues you have identified in achieving fairness in your 
assessment. 

4. Suggest any changes you feel would improve the fairness of your assessment approach. PTO 
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Have you been involved in the delivery and assessment of this course in a traditional classroom 
based face-to-face mode of delivery? 

[ ] Yes [ ] No 

If yes, How different were the assessment approaches for the two delivery models? 

Briefly comment on any differences you experienced in the approach to assessment and the 
assessment tools used for the two delivery models. 
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